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OVERVIEW 


P erformance-based scholarships have two main goals: to give students more money for college 
and to provide incentives for academic progress. They are designed to reduce the financial 
burden on low-income students and help them progress academically by offering financial 
aid contingent upon meeting pre-specified academic benchmarks. The scholarships are 
intended to cover a modest amount of students’ educational costs during the semesters they are 
offered — generally between 15 and 25 percent of students’ unmet financial need, the difference 
between students’ calculated financial need to attend college and the financial aid they are awarded. 
The money is paid directly to students, on top of their existing federal and state need-based financial 
aid, and the students themselves decide how best to use the funds. 

MDRC launched the Performance-Based Scholarship Demonstration in 2008 to evaluate the effec- 
tiveness of these scholarships for as broad a range of low-income students as possible, in a variety of 
settings, and with varying incentive structures. As such, the evaluation includes more than 12,000 
students in institutions across six states to test different performance-based scholarship designs. 
Each program was developed for a different population of students and had a different scholarship 
structure; the scholarship amounts ranged from a few hundred dollars to $1,500 per term, depend- 
ing in part on the benchmarks being tested. Institutions created performance-based scholarship 
programs tailored to what they perceived to be the specific needs of their students, by targeting 
the incentive, academic benchmarks, and in some cases additional services to address those needs. 

Each of the six programs in the demonstration was evaluated using a randomized controlled trial 
— the highest standard of evidence for evaluation research. Students were randomly assigned by 
researchers either to receive only their usual financial aid package and services or to be eligible to 
receive supplemental financial aid and services in the form of a performance-based scholarship, 
contingent upon meeting the given academic benchmarks. 

The results show that these scholarships improved students’ academic progress during the program 
— effects that remained evident several years after the program ended. The effects on students’ aca- 
demic progress appear generally consistent across the different programs and student subgroups. In 
addition, one program targeted high school seniors and succeeded in increasing their matriculation 
in college, and three of the programs reduced students’ dependency on loans. Most important, this 
evaluation finds that the programs modestly increased degree completion, measured after five years. 

These results show that even relatively moderate investments in low-income students’ education can 
have modest but long-lasting impacts on their academic outcomes. These findings may be especially 
relevant to states, institutions, and private scholarship providers seeking purposeful and efficient ways 
to give low-income students additional financial aid that can also help them succeed academically. 
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PREFACE 


P olicymakers in the United States are increasingly focused on expanding the number of col- 
lege graduates, in addition to ensuring broad and equal access to higher education. Rising 
tuition costs, however, pose a considerable challenge to these goals, particularly for low- 
income students, who enroll in and complete college at lower rates than their more affluent 
peers. Financial aid can help, but often it does not cover the full cost of higher education. Although 
the Pell Grant is the main source of federal aid, scholarships also play an important role: State and 
private donors award more than $20 billion annually to undergraduate students. Yet little rigorous 
research has been done to test whether and how scholarships increase college completion rates. 

More than a decade ago, MDRC began evaluating performance-based scholarships, which are designed 
to encourage academic progress and are paid only when students achieve key academic benchmarks, 
such as enrolling in and then earning a pre-specified number of credits. Promising results from an 
early test of a performance-based scholarship program in Louisiana led to a broad demonstration 
study — the Performance-Based Scholarship (PBS) Demonstration — to test the effectiveness of 
this approach, using randomized controlled trials in multiple settings and with varying scholarship 
durations, amounts, and incentives. 

This report marks the culmination of the PBS Demonstration, which included more than 12,000 
students in six different states. The PBS programs generally lasted about a year, and the scholarships 
were designed to cover about 15 to 25 percent of students’ remaining financial need exclusive of other 
aid. The programs produced impacts on academic outcomes that were evident several years after 
the programs ended, in some cases reduced student loans, increased college matriculation in one 
program that targeted high school seniors, and increased students’ use of support services when the 
scholarship was conditioned on the use of those services. Perhaps most important, the scholarship 
programs helped students earn more credits toward their degrees and modestly improved graduation 
rates — and the effects appear consistent across different programs. A variety of performance-based 
scholarship programs have now been effectively implemented in multiple, diverse settings, at both 
the college and the state levels. 

As the focus on college completion intensifies, the results of the PBS Demonstration are encourag- 
ing: Modest financial award programs aimed at encouraging academic progress can help students 
advance toward their degrees and can even make the difference in whether some students graduate. 
Financial aid providers that add to existing federal and state aid programs can build on these results 
by tying scholarship payments both to key steps that help students academically and to important 
markers of academic progress toward earning a degree. 


Gordon L. Berlin 
President, MDRC 
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EXECUTIVE SUMMARY 


P erformance-based scholarships have two main goals: to give students more money for college 
and to provide incentives for academic progress. MDRC launched the Performance-Based 
Scholarship (PBS) Demonstration in 2008 to evaluate the effectiveness of these scholarships 
in a diverse set of states, institutions, and low-income student populations. The evaluation 
includes more than 12,000 students in eight institutions and one intermediary across six states to 
test different performance-based scholarship designs. Each of the programs in the demonstration 
was evaluated using a randomized controlled trial, widely considered to be the most reliable way to 
detect the impact of an intervention. 1 

THE PERFORMANCE-BASED SCHOLARSHIP DEMONSTRATION 

MDRC launched the Performance-Based Scholarship Demonstration in 2008 with anchor funding 
from the Bill & Melinda Gates Foundation and a consortium of other funders: the Helios Education 
Foundation, the Institute of Education Sciences in the U.S. Department of Education, the Joyce 
Foundation, the Kresge Foundation, the NYC Center for Economic Opportunity, the Ohio Department 
of Job and Family Services through the Ohio Board of Regents, the Open Society Foundations, the 
Robin Hood Foundation, and the College Access Foundation of California, now the College Futures 
Foundation. 

Table ES.l presents the design of each program in the PBS Demonstration along with the student 
population that the program targeted. The Opening Doors Louisiana program, an earlier MDRC 
study whose results informed the PBS Demonstration, is included in the table for reference. Each 
program was designed for a different population of students and had a different scholarship structure; 
the institutions’ leaders thought carefully about the needs of their students and designed programs 
accordingly. Across the demonstration, the scholarship amounts ranged from several hundred dollars 
to $1,500 per semester, depending in part on the academic benchmarks being tested and the level 
of students’ financial need. The goal of the evaluation was to test the effectiveness of the programs 
for as broad a range of low-income students as possible, to see whether performance-based scholar- 
ship programs could work in a variety of settings, with a variety of target populations, and with a 
variety of incentive structures. 

While the details differed from state to state, the programs all shared a few key components. Generally, 
students were offered multiple payments over the course of each semester for meeting a pre-specified 
academic benchmark. Each semester, the value of the scholarship payment associated with each 


1. In a randomized controlled trial, study enrollees are randomly assigned either to a program group that is eligible 
to participate in the intervention, or to a control group that is not eligible to participate in the intervention. By 
comparing the outcomes of each group, the impact of the intervention can be estimated. 
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TABLE ES.l Design of the Performance-Based Scholarships in Each State 
Final Report on the Performance-Based Scholarship Demonstration 


OPENING DOORS PERFORMANCE-BASED SCHOLARSHIP DEMONSTRATION 


CHARACTERISTIC 

LOUISIANA 9 

ARIZONA 

CALIFORNIA 

FLORIDA 

NEW MEXICO 

NEW YORK 

OHIO 

Eligible population 

• Age 18-34 

• Parent 

• Family income 
below 200% of 
poverty level 

• Hispanic male 

• Fewer than 45 
credits earned 

• EFC below 5,273 b 

• Age 16-19 

• High school 
seniors applying 
for financial aid 

• Below Cal Grant 
A and C income 
threshold 1 

• Age 18+ 

• In need of 
developmental 
math 

• EFC below 5,273 b 

• Age 17-20 

• Freshmen 

• Pell-eligible 

• Age 22-35 

• Live away from 
parents 

• In need of 
developmental 
education 

• Pell-eligible 

• Age 18+ 

• Parent 

• Zero EFC 

Maximum 
scholarship amount 
per term 

$1,000 

$1,500 

$333 (quarter 
institutions) or 
$500 (semester 
institutions) to 
$1,000 d 

$600 

$1,000 

$1,300 

$600 (quarter 
institutions) or 
$900 (semester 
institutions) 

Scholarship duration 

2 semesters 

3 semesters 

1 term to 2 years 

3 semesters 

4 semesters 

2 full semesters 
and 1 summer 
semester 6 

2 semesters or 3 
quarters 

Maximum amount 

$2,000 

$4,500 

$1,000 - $4,000 

$1,800 

$4,000 

$2,600 - $3,900 

$1,800 


Academic 

Complete 6 

Part-time: 

Complete 6 

Complete a 

Complete 12 

Complete 6 

Part-time: 

benchmarks 

credits or more 

Complete 6-11 

credits or more 

sequence of math 

credits or more 

credits or more 

Complete 6-11 


with a "C" average 

credits with a "C" 

with a "C" average 

courses with a "C" 

(1st semester) 

with a "C" or 

credits with a “C" 


or better 

or better in each 
course 

Full-time: 
Complete 12 
credits or more 
with a "C" or 
better in each 
course 

or better 

or better in each 
course 

or 15 credits 
(subsequent 
semesters) with 
a "C" average or 
better 

better in each 
course 

or better in each 
class 

Full-time: 
Complete 12 or 
more credits with 
a "C" or better in 
each course 


(continued) 
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TABLE ES.l (continued) 


OPENING DOORS PERFORMANCE-BASED SCHOLARSHIP DEMONSTRATION 


CHARACTERISTIC 

LOUISIANA 9 

ARIZONA 

CALIFORNIA 

FLORIDA 

NEW MEXICO 

NEW YORK 

OHIO 

Additional service 
criteria 

Meet with adviser 

Meet with adviser, 
complete tutoring 
and workshop 
requirements 

None 

Complete tutoring 
requirements 

Meet with adviser 

None 

None 

Sample size 

1,019 

1,028 

4,92 1 f 

1,075 

1,081 

1,502 

2,285 


SOURCE: Scholarship designs at each site. 

NOTES: a Opening Doors Louisiana, an earlier MDRC study of a performance-based scholarship program, is included for comparative purposes. 

b The EFC (Expected Family Contribution) is a measure of the amount of money that a family is expected to be able to contribute to a student's education, as calculated 
according to federal guidelines. Students with an EFC of up to 5,273 during the 2010-2011 year were eligible for federal Pell Grants. 

c Cal Grant is a financial aid program funded by the state of California. The awards do not have to be paid back, but to qualify students must fall below certain income and 
asset ceilings. 

d The study in California randomly assigned program group members to one of six scholarship types that varied in amount (from $1,000 total to $4,000 total) and duration 
(from one term to two years). Students could take the award to any degree-granting, accredited institution in the country, and payments were adjusted to reflect the 
institution type (quarter or semester). 

e The study in New York randomly assigned program group members to one of two scholarship types. One type was offered over two semesters only; the other was offered 
over two semesters plus one summer semester. 

•Although there were 5,160 study participants, undocumented immigrant students were excluded from the analysis because of data reliability concerns. Thus, the analysis 
sample was 4,921 participants. 


u> 


benchmark increased over time. Most programs offered a small initial payment and a larger final 
payment each semester. For instance, students at the University of New Mexico program received 
a $250 scholarship payment for registering for the required number of credit-hours, another $250 
scholarship payment for being enrolled in those credit-hours at midterm with a grade point average 
(GPA) of 2.0 or higher, and a final $500 payment for earning those credit-hours with a final GPA of 
2.0 or higher at the end of the semester. The scholarships were designed so that students who earned 
them would get additional aid, but students who did not would not be penalized — they would still 
receive the same amount of Pell Grant and other financial aid that they would have received in the 
absence of the program. 2 In addition, students remained eligible for the scholarship throughout the 
duration of the program. For instance, students who missed a scholarship payment in one semes- 
ter by falling short of the benchmark were still eligible for scholarship payments in all subsequent 
semesters during which the scholarship was offered. 

The performance-based scholarship programs in the PBS Demonstration were successfully imple- 
mented at a variety of institutions, for diverse groups of low-income students. Institutions established 
processes to monitor students’ progress and paid students appropriately. Nearly all students in the 
PBS programs received at least one scholarship payment, and students at every site earned more 
financial aid dollars on average because of the programs. In student surveys, moreover, students 
reported using the money primarily for education-related expenses such as tuition and fees, books, 
or basic living expenses. 


KEY FINDINGS 

• The performance-based scholarship programs produced modest, positive impacts on students’ 
academic progress. The analyses pool up to five years of follow-up data on academic outcomes 
from the PBS programs. The programs showed modest, positive, statistically significant impact 
estimates — the estimated change in outcomes caused by the program, measured by the differ- 
ence between the program and control group outcomes — on credit accumulation in every year, 
as shown in Table ES.2. 3 After four years, students who were eligible to participate in the PBS 
program (the program group) had accumulated, on average, a total of 47.2 credits, compared 
with 45.1 credits earned by students who were not in the PBS program (the control group). The 
estimated impact on credit accumulation is 2.1 credits. 

• The programs modestly increased degree completion. Table ES.3 presents impact estimates on 
degrees earned. After five years, 38.3 percent of students in the program group had completed a 
degree, compared with about 35.1 percent of students in the control group. The estimated impact 
on degree completion is 3.3 percentage points. 


2. Students' loans were sometimes reduced in order to accommodate the additional aid. However, since the 
scholarships were typically intended to cover only 15 to 25 percent of students' unmet need, aid displacement 
other than loans was rarely a concern. Due to the design of the program in California, financial aid package data 
were not collected to assess whether displacement took place there. 

3. A statistically significant impact is one that is unlikely to have occurred by chance alone. 
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TABLE ES.2 Impacts on Cumulative Credits Earned, Pooled 
Final Report on the Performance-Based Scholarship Demonstration 


CUMULATIVE 
CREDITS EARNED 



SAMPLE 

PROGRAM 

CONTROL 

IMPACT 

STANDARD 


YEAR 3 

SIZE 

GROUP 

GROUP 

ESTIMATE 13 

ERROR 

P-VALUE 

1 

6,938 

17.5 

16.2 

1 2 *** 

0.24 

0.00 

2 

6,938 

28.6 

26.7 

-| g*** 

0.46 

0.00 

3 

6,938 

36.4 

34.6 

1.8*** 

0.64 

0.00 

4 

4,835 

47.2 

45.1 

2.1** 

1.04 

0.05 


SOURCES: MDRC calculations using transcript data from Pima Community College, Hillsborough 
Community College, the University of New Mexico, the City University of New York, and the Ohio Board of 
Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

Estimates are weighted to account for the different sample sizes and random assignment ratios in each 
state. 

a Years 1, 2, and 3 each include Arizona, Florida, New Mexico, New York, and Ohio. Year 4 includes New 
Mexico, New York, and Ohio. 

b The impact estimate is the estimated change in outcomes caused by the program, measured by the 
difference between the program and control group outcomes. 


• The programs did not have a substantial impact on persistence (measured each year by whether 
students enrolled in school). The programs produced small impacts on enrollment but did not 
produce the large gains observed in the Opening Doors Louisiana program. For example, in the 
PBS Demonstration, the programs produced an estimated average impact of 1.5 percentage points 
on enrollment in Year 2: That year, 79.3 percent of students in the program group and 77.8 percent 
of students in the control group enrolled. There is no evidence of larger impacts in later years. In 
contrast, the Louisiana program increased enrollment at the beginning of Year 2 by an estimated 
12 percentage points: 49.4 percent of students in the program group and 37.6 percent of students 
in the control group enrolled at that time. 4 


4. See Table 4.3 in Lashawn Richburg-Hayes, Thomas Brock, Allen LeBlanc, Christina Paxson, Cecilia Elena Rouse, and 
Lisa Barrow, Providing More Cash for College: Interim Findings from the Performance-Based Scholarship Demonstration 
in California (New York: MDRC, 2009). 
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TABLE ES.3 Impacts on Degrees Earned, Pooled 
Final Report on the Performance-Based Scholarship Demonstration 


PERCENTAGE WHO 
EARNED A DEGREE 



SAMPLE 

PROGRAM 

CONTROL 

IMPACT 

STANDARD 


YEAR 3 

SIZE 

GROUP 

GROUP 

ESTIMATE 13 

ERROR 

P-VALUE 

1 

11,613 

2.9 

2.8 

0.2 

0.3 

0.61 

2 

11,613 

8.3 

7.4 

0.9* 

0.5 

0.07 

3 

11,613 

14.9 

13.6 

1.4** 

0.7 

0.04 

4 

9,510 

26.4 

25.7 

0.7 

0.9 

0.44 

5 

4,868 

38.3 

35.1 

3.3** 

1.4 

0.02 


SOURCES: MDRC calculations using transcript data from the National Student Clearinghouse, Pima 
Community College, Hillsborough Community College, the University of New Mexico, the City University 
of New York, and the Ohio Board of Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

Estimates are weighted to account for the different sample sizes and random assignment ratios in each 
state. 

a Years 1, 2, and 3 each include Arizona, California, Florida, New Mexico, New York, and Ohio. Year 4 
includes California, New Mexico, New York, and Ohio. Year 5 includes New Mexico, New York, and Ohio. 

b The impact estimate is the estimated change in outcomes caused by the program, measured by the 
difference between the program and control group outcomes. 


• The scholarship programs varied along multiple dimensions, but they appear to consistently 
help students progress academically. The PBS programs showed little evidence of variation in 
impacts across multiple dimensions. The programs appear to work for varying scholarship pro- 
gram designs, in different states and academic settings. 

• The scholarships worked for a variety of low-income students with different characteristics, 
including at-risk groups that traditionally perform poorly. The programs were similarly effec- 
tive for a wide variety of student groups, including younger and older students, men and women, 
and Hispanic and non-Hispanic students. 

• Evidence from the study in California suggests that offering students a scholarship in their 
senior year of high school increases enrollment in the first year of college. The program in 
California was the only one to offer scholarships to high school students. The evidence suggests 
that the California program produced the strongest impacts on enrollment in the first semester 
of college, primarily for students who attended community colleges. 
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• In sites where student services accompanied the performance-based scholarship, the use of 
these services increased markedly as a result of the program. At Pima Community College in 
Arizona, Hillsborough Community College in Florida, and the University of New Mexico, stu- 
dent services such as tutoring, advising, and workshops were integrated in various ways with the 
award. Students’ likelihood of using these services increased because of the scholarship program, 
which offered incentives to do so. 

• The performance-based scholarships increased students’ total financial aid, even when helping 
to reduce their dependency on loans. The PBS program consistently increased the average amount 
of financial aid that students in the program group received, ranging from around $500 (at Lorain 
County Community College, Owens Community College, and Sinclair Community College in 
Ohio) to $2,200 (at Borough of Manhattan Community College and Hostos Community College 
in New York). Three of the programs also reduced students’ loans during the time that they were 
offered a scholarship, with loan reductions ranging from about $330 to $600. 

• The scholarships cost additional money but did not increase the average cost per degree. After 
five years of follow-up, the cost per degree for students in the program group and the control 
group is about the same. 


CONCLUSIONS AND LESSONS FROM THIS DEMONSTRATION 

This evaluation suggests that the performance-based scholarship programs accomplished their two 
main goals: The programs helped low-income students pay for college and helped them make greater 
academic progress toward their degrees. These results are also promising in the context of existing 
research on financial aid, which generally finds small positive impacts or no evidence of impacts 
for enrollment or credit accumulation, and less frequently finds impacts on graduation. 5 A key les- 
son for scholarship providers and policymakers at the federal, state, and local levels is that these 
programs did not depend on rigid adherence to a scholarship structure, but instead were guided by 
three core principles: 

1. Scholarship payments were made in increments during a semester and were contingent on 
behaviors associated with greater academic success, such as enrollment in a certain number of 
credits and maintaining that course load over the semester. 


5. See, for example, Joshua Angrist, Daniel Lang, and Philip Oreopoulos, "Incentives and Services for College 

Achievement: Evidence from a Randomized Trial," American Economic Journal: Applied Economics 1, 1 (2009): 1-28; 
Melissa Binder, Kate Krause, Cynthia Miller, and Oscar Cerna, "Providing Incentives for Timely Progress Toward 
Earning a College Degree: Results from a Performance-Based Scholarship Experiment," MDRC Working Paper (New 
York: MDRC, 2015); Sara Goldrick-Rab, Douglas N. Harris, Robert Kelchen, and James Benson, Need-Based Financial 
Aid and College Persistence: Experimental Evidence from Wisconsin (Madison: Wisconsin Center for Educational 
Research, 2012); and Judith Scott-Clayton, "On Money and Motivation: A Quasi-Experimental Analysis of Financial 
Incentives for College Achievement," The Journal of Fluman Resources 46, 3 (2011): 614-646. See Reuben Ford, 
Douwere Grekou, Isaac Kwakye, and Claudia Nicholson, Future to Discover: Fourth Year Post-Secondary Impacts Report 
(Ottawa, Ontario: Social Research and Demonstration Corporation, 2014), for an example of a related program that 
shows evidence of graduation impacts in Canada. 
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2. Scholarship payments that were made at the end of each semester were tied to important bench- 
marks associated with graduation (such as receiving a minimum grade in a certain number of 
credits over the semester) while also providing funds to support students’ education in the next 
semester. 

3. When program design included additional student support services (such as advising or tutor- 
ing), the programs required that students use these services in order to earn the scholarships. 
This requirement improved participation in these services substantially. 

The program designs in each state were created collaboratively with key stakeholders, who pro- 
vided valuable insights related to the target populations for the scholarships, the benchmarks, and 
the student services, if any, that should be incorporated into the design. The flexible nature of the 
programs makes it possible for other scholarship providers — at the state or local level, public or 
private — to adapt the design to meet their own priorities. 6 

This study does not, however, suggest that all financial aid should be restructured to encourage per- 
formance — especially in regard to the federal Pell Grant program. The Pell Grant is generally the 
foundation of a student’s aid package and can be much larger than the scholarship amounts studied 
here, which were paid to students in addition to any Pell dollars and other financial aid they already 
received. Changing disbursement criteria for Pell could alter behavior in ways very different from 
the PBS Demonstration results, and could negatively affect enrollment, since Pell is typically paid 
at the beginning of the semester and students may anticipate having those funds at the time they 
enroll in school. Providing students less financial aid up front could reduce their ability to cover 
tuition and fees at many colleges. 7 

The findings do suggest, however, possibilities for restructuring state and private aid, and any ad- 
ditional federal aid that might be provided in the future. State and private donors contribute more 
than $20.1 billion in scholarships to undergraduate students, and some of this aid may be more ef- 
fectively offered as performance-based scholarships. 8 For the 2012-2013 academic year, 19 percent 
of all state-sponsored aid was reported as merit-based — typically based on students’ high school 
performance, not their college performance. Students who receive this aid, however, may be likely 


6. For more information on this topic, see Rashida Welbeck, Michelle Ware, Oscar Cerna, and Ireri Valenzuela, Paying 
It Forward: A Technical Assistance Guide for Developing and Implementing Performance-Based Scholarships (New York: 
MDRC, 2014). 

7. In 2010-2011, the maximum Pell Grant amount was $5,500. That same year, the median tuition and fees for all public 
institutions was $4,632. For all public four-year institutions it was $6,780, and for all public two-year institutions it 
was $2,537. These figures represent tuition and fees over a full year; each semester would cost roughly half that. See 
National Center for Education Statistics, "Average Undergraduate Tuition, Fees, Room, and Board Rates for Full-Time 
Students in Degree-Granting Postsecondary Institutions, by Percentile, Control and Level of Institution: Selected 
Years, 2000-01 Through 2012-13" (Washington, DC: NCES, 2011), Table 330.30 (website: http://nces.ed.gov). 

8. Sandy Baum, Diane Cardenas Elliott, and Jennifer Ma, Trends in Student Aid 2014 (Washington, DC: The College Board, 
2014). 
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to succeed in college without it. 9 Reallocating merit-based aid to low-income students through 
performance-based scholarship programs could be an effective strategy to help these students make 
greater academic progress. It could also help programs offer money to more students or increase the 
size of scholarships. This is because students in the programs were offered the opportunity to earn 
more scholarships, but not all students earned them: The programs did not pay those students the full 
amount that was offered. Some of this saved money, for example, could be offered to other students. 

Some states have also used existing funds creatively to promote greater academic progress through 
college for low-income populations. Both the original performance-based scholarship program 
launched in Louisiana as part of the Opening Doors Demonstration and the Ohio PBS program 
described in this report used state Temporary Assistance for Needy Families (TANF) funds to offer 
these scholarships to low-income parents who were attending college. More recently, new legisla- 
tion in Indiana includes provisions to give financial bonuses to some low-income students if they 
meet certain academic benchmarks. Students in the Frank O’Bannon program, for example — a 
need-based financial aid program — can earn an additional $1,300 in aid if they complete 39 credits 
annually. 10 California is currently considering a similar program. * 11 

The results presented here demonstrate that performance-based scholarship programs have consis- 
tently helped students make modest gains in their academic progress and even helped some students 
graduate. The programs provide a flexible design that can be implemented widely, and they can be 
structured to meet the needs of diverse student populations. This evaluation shows that performance- 
based scholarships are an important tool for colleges, states, and scholarship providers who work to 
improve outcomes for low-income students. 


9. National Association of State Student Grant and Aid Programs (NASSGAP), 44th Annual NASSGAP Survey Report, 
2012-2013 Academic Year (n.d.), website: www.nassgap.org; Sandy Baum, David W. Breneman, Matthew M. Chingos, 
Ronald G. Ehrenberg, Pamela Fowler, John Hayek, Donald E. Heller, Allison G. Jones, David A. Longanecker, Tim 
Nesbitt, Judith Scott-Clayton, Sarah E. Turner, Jane V. Wellman, and Grover "Russ" Whitehurst, "Beyond Need and 
Merit: Strengthening State Grant Programs" (Washington, DC: The Brookings Institute, 2012). 

10. Indiana Commission for Higher Education, Reforming Student Financial Aid to Increase College Completion: Early 
Progress Resulting from Indiana House Enrolled Act 1348 (Indianapolis: Indiana Commission for Higher Education, 
2015). 

11. For more information about this program, see California State Senate Majority Caucus, "Fact Sheet on SB 15: A Plan 
for Higher Education in California" (Sacramento: California State Senate Majority Caucus, 2014). 


Executive Summary 


E S - 9 



INTRODUCTION 


Performance-based scholarships have two main goals: to give students more money for college and 
to provide incentives for academic progress. MDRC launched the Performance-Based Scholarship 
(PBS) Demonstration in 2008 to evaluate the effectiveness of these scholarships in a diverse set of 
states, institutions, and low-income student populations. Since then, MDRC has worked with more 
than 12,000 students in eight institutions and one intermediary across six states to test different 
performance-based scholarship designs. 

Performance-based scholarships are designed to help low-income students pay for college by offering 
them financial aid contingent upon meeting pre-specified academic benchmarks. The scholarships 
are intended to cover a modest amount of students’ educational costs during the semesters they are 
offered — generally between 15 and 25 percent of students’ unmet financial need, or the difference 
between students’ calculated financial need to attend college and the financial aid they are awarded. 
The money is paid directly to students, on top of their existing federal and state need-based financial 
aid, and the students themselves decide how best to use the funds. Students can spend the money on 
educational expenses, such as textbooks and tuition; on personal expenses, such as transportation 
or child care; or on other items. Unlike merit-based financial aid, which is offered based on past 
academic performance, these scholarships offer students money for future performance. By making 
the money conditional upon fulfillment of key academic benchmarks, such as full-time enrollment 
and credit accumulation with a “C” average or better, the programs seek to support and encourage 
students’ academic progress at the same time. 

This report finds that performance-based scholarship programs consistently produced modest, posi- 
tive impacts on key indicators of students’ academic success over time. Students in the programs were 
more likely to meet key academic benchmarks and to use student services when they were tied to 
the scholarship payments, leading to gains in enrollment, credit accumulation, and degree comple- 
tion. The demonstration also found that performance-based scholarships can be implemented in a 
wide variety of institutional contexts for diverse populations of low-income students. Offering these 
scholarships to students increased their total financial aid, even while reducing their dependency 
on loans. 


Background 

Over the past decade, tuition and fees at public colleges and universities across the United States 
have risen much faster than the rate of inflation. Since the 2000-2001 school year, published average 
tuition and fees for community college students have increased from $2,264 to $3,347 in the 2014- 
2015 academic year — an increase of nearly 50 percent. For students at public four-year institutions, 
the increase has been even higher, rising from $4,837 to $9,139, or 88 percent. 1 

College enrollment in the United States has grown steadily over the past several decades, but many 
low-income students still do not attend college. In 2009, only 55 percent of high school graduates 


1. Prices are converted to 2014 dollars. For more information, see College Board (2015). 
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from low-income families went to college, compared with 67 percent and 84 percent of those from 
middle-income and highly affluent families, respectively. 2 Success rates are also quite low, especially 
among low-income students and students of color. Research by the U.S. Department of Education 
found that only 35 percent of incoming students at community colleges and 64 percent of incoming 
students at four-year institutions earn a certificate or degree from any college or university after 
six years. 3 

While financial aid allows many students and their families to pay less than the published prices 
for college tuition, that aid is often not enough to cover the full cost of attendance, which includes 
additional expenses such as textbooks, transportation, and living expenses — all of which have 
also increased in price. One study found that the vast majority of low-income students face unmet 
financial need: Among community college students with documented financial need, 80 percent 
still have unmet need after all financial aid has been awarded. Among this group are the 40 per- 
cent of community college students whose family incomes are so low that their Expected Family 
Contribution (EFC) is zero. 4 

Private scholarships, which award over $10 billion to undergraduate students annually, are one 
resource that can help cover students’ unmet need. 5 A community college student who receives 
significant private scholarship money, however, is the exception, not the rule. 6 Private scholarships 
are traditionally merit-based and awarded to students based on their past academic success — often 
characterized by a good high school grade point average (GPA). This merit-based aid may not be 
within reach for the average low-income community college student. 7 

Several studies, however, show that financial aid correlates with college enrollment and persistence, 
suggesting that giving low-income students additional financial aid might help them academically. 8 
Need-based grant aid, in particular, has been estimated to have a positive impact on credit accumula- 
tion and college persistence to the second year. 9 A review of the literature also notes that financial 
aid may affect students differently depending on their level of financial need. 10 Few studies, however, 
have used randomized controlled trials to examine the causal impact of innovative financial aid 


2. Baum, Kurose, and McPherson (2013). 

3. Walton Radford, Berkner, Wheeless, and Shepherd (2010). 

4. The Institute for College Access and Success (2009). The Expected Family Contribution is a measure of a student's 
and/or family's ability to contribute toward the cost of college and is calculated according to a formula established 
by federal law. Elements of the formula may include the family's taxed and untaxed income, assets, and benefits, 
such as unemployment and Social Security; the size of the family; and the number of family members enrolled in 
postsecondary education in the given year. 

5. Baum, Cardenas Elliott, and Ma (2014). 

6. Baum and Ma (2014). 

7. Huelsman and Cunningham (2013). 

8. St. John, Andrieu, Oescher, and Starkey (1994); Avery and Kane (2004); Deming and Dynarski (2009); Cornwell, 
Mustard, and Sridhar (2006); Dynarski and Scott-Clayton (2013). 

9. Dynarski (2003); Castleman and Long (2012); Bettinger (2004); Bettinger (2010). 

10. Welbeck, Diamond, Mayer, and Richburg-Hayes (2014). 
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structures on long-term student achievement, particularly graduation, 11 and quasi-experimental 
studies have found that aid programs have mixed results. One such quasi-experimental study found 
that a merit-based aid program in Georgia produced large increases in graduation rates. 12 Follow-up 
studies that also used quasi-experimental designs, however, found no statistically significant differ- 
ences in degree attainment for students receiving the Georgia scholarships. 13 

Studies that tested state scholarship programs with regression discontinuity designs also found mixed 
results. 14 One found that Tennessee’s scholarship program had no effect on students’ educational 
attainment, 15 while another found an impact on students’ four-year graduation rates of 9 percentage 
points in West Virginia. 16 While differences in the student samples may explain these conflicting 
findings, 17 there may be other explanations, and the impact of such programs remains unclear. 

MDRC’s Opening Doors Demonstration in Louisiana, launched in 2004, was one of the first stud- 
ies to rigorously test a need-based financial aid program for low-income students on a large scale, 
using a randomized controlled trial. 18 Using this research design alleviates concerns about whether 
students receiving financial aid differ in important ways from those not receiving aid, which could 
confound results, because random assignment ensures that no systematic factors determine whether 
students are receiving aid. 

The Opening Doors Louisiana program offered low-income students with children a performance- 
based scholarship of $1,000 per semester for two semesters in exchange for enrolling in college at 
least half-time and maintaining a “C” average or better each semester. 19 MDRC’s evaluation of the 
program found that students who were eligible for the scholarship were more likely to enroll in 
college full time and persist from semester to semester, compared with students in a control group 
who were not eligible for the scholarship. In fact, the program was estimated to increase enrollment 
in the second semester by 15 percentage points — a larger impact than MDRC had seen in other 
studies at the time, and one that inspired hope of impacts on graduation. Students eligible to receive 
the performance-based scholarship also accumulated an average of 2.3 more credits than students 


11. Some of the exceptions are Angrist, Lang, and Oreopoulos (2009); Binder, Krause, Miller, and Cerna (2015); Ford, 
Grekou, Kwakye, and Nicholson (2014); Goldrick-Rab, Harris, Kelchen, and Benson (2012); and Richburg-Hayes et al. 
(2009). Castleman and Long (2012) identify this gap in the literature as resulting from the challenges in separating 
student characteristics from financial aid receipt. For example, low-income students who qualify for need-based 
financial aid may be more academically underprepared in addition to having higher levels of financial need 
compared with their higher-income peers. 

12. Dynarski (2000). 

13. Sjoquist and Winters (2012). 

14. Regression discontinuity analysis is a quasi-experimental design that can be used to estimate program impacts by 
comparing outcomes for individuals who fall just above and just below a particular eligibility cutoff. 

15. Bruce and Carruthers (2011). 

16. Scott-Clayton (2011). 

17. Binder, Krause, Miller, and Cerna (2015). 

18. Other randomized controlled trials looking at financial aid programs have been conducted since 2004, including 
those listed in footnote 11. 

19. Information in this paragraph is discussed in more detail in Richburg-Hayes et al. (2009). 
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in the control group during the first year. The devastation wreaked by Hurricane Katrina in 2005, 
however, disrupted students’ education, and plans for a longer-term follow-up to look for impacts 
on degree receipt were discontinued. 


The Performance-Based Scholarship Demonstration 

Following the positive results in Louisiana, MDRC launched the Performance-Based Scholarship 
Demonstration in 2008 with anchor funding from the Bill & Melinda Gates Foundation and a consor- 
tium of other funders: The Helios Education Foundation, The Institute of Education Sciences in the 
U.S. Department of Education, The Joyce Foundation, The Kresge Foundation, The NYC Center for 
Economic Opportunity, The Ohio Department of Job and Family Services through the Ohio Board 
of Regents, The Open Society Foundations, The Robin Hood Foundation, and The College Access 
Foundation of California, now The College Futures Foundation. The PBS Demonstration aimed to 
test whether the promising results in Louisiana could be replicated in other states, institutions, and 
contexts and with other low-income student populations. 

Table 1 presents the designs of each program in the PBS Demonstration along with the student 
population the program targeted. The Opening Doors Louisiana program is included for reference. 
Each program was designed for a different population of students and had a different scholarship 
structure; the scholarship amounts ranged from a few hundred dollars per term to $1,500 per term, 
depending in part on the benchmarks being tested and the level of students’ financial need. The goal 
of the evaluation was to test the effectiveness of the programs for as broad a range of low-income 
students as possible, to see whether performance-based scholarship programs could work in a variety 
of settings, with different target populations, and with varying incentive structures. Institutions 
took advantage of this flexibility to create performance-based scholarship programs tailored to what 
they perceived to be the specific needs of their students, by targeting the incentives, benchmarks, 
and in some cases additional services to address those needs. 20 

In Arizona, the performance-based scholarship program operated at Pima Community College. 
The program focused on creating academic supports to help Latino males succeed and provided a 
financial incentive to encourage students to go to school full time. School administrators at Pima 
were concerned that these students were enrolling in too few credits and not taking advantage of 
the student support services available on campus. Students in the program were eligible to receive 
up to $1,500 per semester for three semesters for passing their courses with a “C” grade or better. 
Pima’s program targeted Latino male students exclusively and provided incentives for attending 
advising and tutoring sessions, on top of the incentives for passing their courses. In addition, the 
program ran group sessions called Platicas (Spanish for “conversations”) in which students could 
discuss obstacles and issues unique to their lives. Students had to attend one Platica per semester 
to qualify for the full payment. 


20. More information about implementing these scholarships can be found in Welbeck, Ware, Cerna, and Valenzuela 
(2014). 
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TABLE 1 Design of the Performance-Based Scholarships in Each State 
Final Report on the Performance-Based Scholarship Demonstration 


in 


OPENING DOORS 


PERFORMANCE-BASED SCHOLARSHIP DEMONSTRATION 


CHARACTERISTIC 

LOUISIANA 9 

ARIZONA 

CALIFORNIA 

FLORIDA 

NEW MEXICO 

NEW YORK 

OHIO 

Eligible population 

• Age 18-34 

• Parent 

• Family income 
below 200% of 
poverty level 

• Flispanic male 

• Fewer than 45 
credits earned 

• EFC below 5,273 b 

• Age 16-19 

• High school 
seniors applying 
for financial aid 

• Below Cal Grant 
A and C income 
threshold 1 

• Age 18+ 

• In need of 
developmental 
math 

• EFC below 5,273 b 

• Age 17-20 

• Freshmen 

• Pell-eligible 

• Age 22-35 

• Live away from 
parents 

• In need of 
developmental 
education 

• Pell-eligible 

• Age 18+ 

• Parent 

• Zero EFC 

Maximum 
scholarship amount 
per term 

$1,000 

$1,500 

$333 (quarter 
institutions) or 
$500 (semester 
institutions) to 
$1,000 d 

$600 

$1,000 

$1,300 

$600 (quarter 
institutions) or 
$900 (semester 
institutions) 

Scholarship duration 

2 semesters 

3 semesters 

1 term to 2 years 

3 semesters 

4 semesters 

2 full semesters 
and 1 summer 
semester 6 

2 semesters or 3 
quarters 

Maximum amount 

$2,000 

$4,500 

$1,000 -$4,000 

$1,800 

$4,000 

$2,600 - $3,900 

$1,800 

Academic 

Complete 6 

Part-time: 

Complete 6 

Complete a 

Complete 12 

Complete 6 

Part-time: 

benchmarks 

credits or more 

Complete 6-11 

credits or more 

sequence of math 

credits or more 

credits or more 

Complete 6-11 


with a "C" average 

credits with a "C" 

with a "C" average 

courses with a "C" 

(1st semester) 

with a "C" or 

credits with a "C" 


or better 

or better in each 
course 

Full-time: 
Complete 12 
credits or more 
with a "C" or 
better in each 
course 

or better 

or better in each 
course 

or 15 credits 
(subsequent 
semesters) with 
a “C" average or 
better 

better in each 
course 

or better in each 
class 

Full-time: 
Complete 12 or 
more credits with 
a "C" or better in 
each course 
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(continued) 





OPENING DOORS 


PERFORMANCE-BASED SCHOLARSHIP DEMONSTRATION 


CHARACTERISTIC 

LOUISIANA 3 

ARIZONA 

CALIFORNIA 

FLORIDA 

NEW MEXICO 

NEW YORK 

OHIO 

Additional service 
criteria 

Meet with adviser 

Meet with adviser, 
complete tutoring 
and workshop 
requirements 

None 

Complete tutoring 
requirements 

Meet with adviser 

None 

None 

Sample size 

1,019 

1,028 

4,92 T 

1,075 

1,081 

1,502 

2,285 


SOURCE: Scholarship designs at each site. 

NOTES: a Opening Doors Louisiana, an earlier MDRC study of a performance-based scholarship program, is included for comparative purposes. 

b The EFC (Expected Family Contribution) is a measure of the amount of money that a family is expected to be able to contribute to a student's education, as calculated 
according to federal guidelines. Students with an EFC of up to 5,273 during the 2010-2011 year were eligible for federal Pell Grants. 

c Cal Grant is a financial aid program funded by the state of California. The awards do not have to be paid back, but to qualify students must fall below certain income and 
asset ceilings. 

d The study in California randomly assigned program group members to one of six scholarship types that varied in amount (from $1,000 total to $4,000 total) and duration 
(from one term to two years). Students could take the award to any degree-granting, accredited institution in the country, and payments were adjusted to reflect the 
institution type (quarter or semester). 

e The study in New York randomly assigned program group members to one of two scholarship types. One type was offered over two semesters only; the other was offered 
over two semesters plus one summer semester. 

Although there were 5,160 study participants, undocumented immigrant students were excluded from the analysis because of data reliability concerns. Thus, the analysis 
sample was 4,921 participants. 


In California, the program targeted low-income high school seniors planning to attend college 
who went to a Cash for College financial aid workshop. The resulting study population was over 
60 percent Latino. Students were randomly assigned into seven groups: Five groups were offered 
performance-based scholarships of varying amounts and durations with enrollment and GPA re- 
quirements; one group was offered a traditional scholarship without performance criteria; and one 
group was a control group that was not offered a scholarship. The program allowed students to use 
their scholarship at any accredited two-year or four-year institution of their choosing, whether in 
California or across the country. This large-scale program operated at the state level rather than at 
individual institutions and gave students broad flexibility to choose institutions. The goal of placing 
students into different types of scholarship groups was to examine how varied scholarship durations 
and amounts affected student behavior. 


“THE INSTRUCTORS AND THE . . . STAFF [HAVE] GIVEN ME 
A SENSE OF BELONGING, A SENSE THAT THEY DO CARE, 
AND THEY’RE WILLING TO STRETCH OUT FOR SOMEONE 
THAT’S BARELY NAVIGATING THEIR WAY AROUND. . . . 

THEY . . . KIND OF GIVE YOU A PATH TO FOLLOW AND 
THAT’S BETTER THAN JUST WINGING IT. I DIDN’T HAVE 
THAT SUPPORT WHEN I WENT TO SCHOOL BEFORE.” 

— A STUDENT IN ARIZONA 

In Florida, school administrators at Hillsborough Community College identified a specific sequence 
of developmental math classes as a roadblock to student success. Hillsborough’s scholarship program 
was designed to encourage students to enroll in and complete their developmental math course se- 
quences. Reflecting the diversity of the school, the sample of students in this particular study was 
almost evenly split among white, black, and Latino students. The students were an average age of 27 
years old at the time of random assignment, and many were parents, signifying that a large proportion 
of students were not of traditional college-going age. Students were offered a maximum of $600 per 
semester for three semesters for enrolling in and passing the developmental math course into which 
they had been placed. Each semester, they needed to enroll in and then pass the next course in the 
sequence to earn their scholarship payments. Program administrators at Hillsborough Community 
College also hoped to increase the number of students taking advantage of math support services 
and tutoring available on campus, so a service component was included in the scholarship: In order 
for students to receive the second payment each semester, they needed to complete a set number of 
hours in the campus Math Lab, a math learning center offering tutoring, group work, computerized 
practice sets, and more. 

At the University of New Mexico — the only four-year institution in the demonstration to run a 
performance-based scholarship program — program designers wanted to encourage students of 
traditional college-going age to attempt and complete enough credits each semester to increase 
their likelihood of graduating on time. The performance-based scholarship program was structured 
to pay students for enrolling in and completing 12 credits in their first semester and 15 credits in 
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each subsequent semester, offering students a total of $1,000 per semester for four semesters. This 
program was predominantly composed of Latino students between the ages of 17 and 18 years old. 

In New York, the program was designed to help nontraditional students with developmental education 
needs stay in college. The students at these colleges were an average age of 27 years old, and almost 
half were parents. More than 90 percent were students of color. Students were offered a maximum 
scholarship of $1,300 per semester for two semesters. The two New York institutions, Borough of 
Manhattan Community College and Hostos Community College, identified summer enrollment as 
a way for students to accumulate more credits, so some of the students were also randomly assigned 
to receive an additional scholarship offer for summer enrollment. 

Finally, in Ohio, the program targeted low-income parents. Here, three schools participated: Lorain 
County Community College, Owens Community College, and Sinclair Community College. The 
vast majority of these student-parents were female, and they were an average age of 30 years old; 
almost two-thirds of these students were from households receiving public benefits. A differential 
full-time and part-time scholarship was created to help encourage students to enroll full time in 
school, while also supporting students who could attend only part time. Students who enrolled in 
12 credits or more were eligible for a maximum award of $900 per semester, while students who 
enrolled in 6 to 11 credits were eligible for up to $450 per semester. At each institution, whether it 
operated on a semester or a quarter system, students were eligible for a maximum award of $1,800 
over the course of the entire program. 

Across these states, the scholarship programs’ designs differed, but they shared a few key compo- 
nents. Generally, students were offered multiple payments each term for meeting a pre-specified 
academic benchmark. Over the course of each semester, the value of the scholarship payment associ- 
ated with the benchmarks increased, with most programs offering a small payment at the beginning 
of the semester and a larger payment at the end of the semester, so that the total amount offered 
covered about 15 to 25 percent of students’ unmet need for that semester. For instance, students at 
the University of New Mexico received a $250 scholarship payment for registering for the required 
number of credit-hours, another $250 scholarship payment for being enrolled in those credit-hours 
at midterm with a GPA of 2.0 or higher, and a final $500 payment for earning those credit-hours 
with a GPA of 2.0 or higher at the end of the semester. An important component of the design is 
that scholarship payments are made directly to students, who can use the money as they like, and 
the money is provided in addition to Pell Grants and any need-based state aid that students might 
receive. Though the scholarships were designed so that students who earned them would get ad- 
ditional aid, students who did not receive the scholarship were not penalized: All students in the 
study still received the same amount of Pell and other financial aid that they would have received 
in the absence of the program. 21 


21. Students' loans were sometimes reduced in order to accommodate the additional aid. However, since the 
scholarships were typically intended to cover only 15 to 25 percent of students' unmet need, aid displacement 
other than loans was rarely a concern. Due to the design of the program in California, financial aid package data 
were not collected to assess whether displacement took place there. 
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The goal of all the programs was to reduce financial burdens on low-income students, to pro- 
vide incentives for good academic progress, and generally to help students succeed academically. 
Performance-based scholarships are theorized to work by offering students an incentive to make 
positive changes in their behaviors associated with academic success — for example, by increasing 
study time, encouraging students to work harder at their courses, and helping them to attempt and 
earn additional credits. As a result of the programs, students are expected to feel increased confi- 
dence in their ability to succeed and to feel less stress about money while in college. Additionally, 
students who earn scholarship dollars may have more money to put toward their college expenses in 
the subsequent semesters, which might lead students to spend less time at a job or to prioritize school 
over employment at key points in the semester, such as during midterm exams. At the institutions 
where a service component was tied to scholarship receipt, students were expected to benefit from 
that service in addition to the money they received. Tutoring, for example, could help students do 
better in class, and academic advising could help them make better academic decisions. 


“I GOT OUT OF HIGH SCHOOL. I HAD TWO BABIES. I 
SUPPORTED MYSELF WITH WAIT RES SING, AND I JUST 
GOT FED UP WITH IT. . . . SO I DECIDED TO COME BACK TO 
SCHOOL TO GET A MORE STABLE JOB . . . TO BE ABLE TO 
HAVE MONEY . . . AND THE THINGS THAT WE WANTED.” 

— A STUDENT IN OHIO 


These short-term outcomes could help students stay in college and shorten the amount of time it 
takes to attain a degree. Ultimately, these behavioral changes are expected to lead to an increased 
likelihood of graduation, better employment prospects, and higher earnings associated with a col- 
lege credential. 

All of the institutions in the demonstration were purposeful in the design of their scholarship program, 
focusing on the perceived needs of their students and targeting the incentives and benchmarks to 
those needs. While no scholarship program offered enough to fully cover students’ unmet financial 
need, they did seek to offer an amount large enough to influence students’ behavior. 


Research Questions and Design 

The PBS Demonstration addresses the following research questions: 

• Can performance-based scholarships be implemented widely, in diverse settings, for diverse 
groups of students? 

• Do performance-based scholarships affect important academic outcomes, as measured by enroll- 
ment, the number of credits students earn, and attainment of degrees and certificates? 
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• Do the impacts of performance-based scholarships vary for different scholarship programs or 
for different types of students? 

• What is the cost to implement performance-based scholarship programs? 

• Are performance-based scholarship programs cost-effective? 

MDRC evaluated the Performance-Based Scholarship Demonstration using a random assignment 
research design. Widely considered to be the gold standard in social science research, random assign- 
ment constructs two groups of study enrollees — a program group and a control group — who are, 
on average, similar at the outset of the study in both their observable characteristics (for example, 
ethnicity, gender) and their unobservable ones (such as motivation and personal obstacles). As a 
result, any subsequent differences in outcomes between the two groups can be confidently attributed 
to the performance-based scholarship program. In the PBS Demonstration, eligible students who 
consented to join the study were randomly assigned to either a program group, in which they were 
eligible to receive performance-based scholarships, or to a control group, in which they received 
their usual financial aid package and all standard services their institution offered but no additional 
performance-based scholarship. 

This report analyzes pooled data from each of the sites included in the PBS Demonstration and 
synthesizes findings across the studies to present conclusions drawn from the entire demonstration 
related to feasibility, efficacy, and cost-effectiveness. It provides long-term follow-up data for all sites, 
including up to five years of follow-up for academic outcomes. The PBS Demonstration adds to a 
growing knowledge base about the effectiveness of scholarships in improving students’ academic 
outcomes. This large demonstration allows analysis of diverse student populations receiving differ- 
ent types of scholarship offers. This report shows that while the performance-based scholarships 
covered just a portion of students’ overall unmet financial need, they produced consistently positive, 
modest impacts on academic outcomes. 


The balance of this report presents the data sources for the evaluation and demographics for the 
students in the study, followed by the findings on implementation, impacts, and costs. The report 
concludes with the implications of these findings for states, institutions, and scholarship providers. 


DATA SOURCES FOR THIS REPORT 

The evaluation of performance-based scholarships in this report is based on the data sources de- 
scribed below. 

• Baseline data. All students in the PBS Demonstration completed a baseline information ques- 
tionnaire before random assignment, detailing a range of demographic and other background 
information. These data are used to describe the sample and to identify students for subgroup 
analyses. Appendix Table A.l shows the baseline information for the sample students by state. 
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• Financial aid information. MDRC received financial aid data for all sites except California for 
at least the first full year after random assignment took place. These data include information 
on the performance-based scholarship amounts awarded to or actually received by students, as 
well as federal Pell Grants, student loans, and other financial supports unique to the individual 
sites. The colleges in the study provided financial aid data directly to the study team. As a result, 
California data were not obtained because students in that state were able to take their scholar- 
ship to any accredited two-year or four-year school. 

• Operational site visits, field research, and student surveys. As a complement to the administra- 
tive data collected on students, MDRC conducted interviews and focus groups with students in 
both the program and control groups at several of the sites. In addition, surveys were conducted 
with the students in California, at the Ohio colleges, at Pima Community College in Arizona, 
and at Hillsborough Community College in Florida. 

• Student records data. Student records follow-up data were available for varying lengths of time 
in different sites. Table 2 shows the data source and the follow-up periods available for each of 
the academic outcomes included in this report. For all states except California, data from the 
National Student Clearinghouse supplemented the data sets for enrollment and degrees earned. 22 
The Clearinghouse, a nonprofit organization, collects and distributes enrollment, degree, and 
certificate data from more than 3,600 colleges that (combined) enroll 98 percent of the nation’s 
college students. 23 

• Cost data. All of the institutions provided expenditure data on their respective programs. These 
data were used to estimate the cost of the program at each site. To estimate the cost of college 
services used by students, data on credits attempted by students and annual expenditures for 
all of the institutions in the demonstration were collected from the Integrated Postsecondary 
Education Data System (IPEDS). The Higher Education Price Index was used to update all cost 
data to reflect 2014 dollars to account for inflation over the course of this study. 

The initial 2004 study of performance-based scholarships, the Opening Doors study in Louisiana, 

is excluded from much of the remainder of this report. Due to the destruction caused by Hurricane 

Katrina, researchers did not examine that program’s long-term impacts. 24 


22. California's analysis includes five different types of performance-based scholarships. Aalysis of a sixth scholarship 
type, which was not performance-based, is not included in this report. 

23. Coverage reflects information published by the National Student Clearinghouse (2015). Coverage may have 
changed since students' records data were obtained in November of 2014. When there was overlap in the data 
coverage, the individual site data were used to identify academic outcomes at the schools of random assignment 
and the Clearinghouse data were used to identify academic outcomes at other schools. In situations where there 
was an additional year of Clearinghouse data beyond the overlap period, the Clearinghouse data were used to 
identify academic outcomes at both the school of random assignment and any other school. For New York and 
Ohio, this approach resulted in an additional year of follow-up. For California, Clearinghouse data were used as the 
primary data source for the full follow-up period (four years). 

24. Barrow, Richburg-Flayes, Rouse, and Brock (2014). 
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TABLE 2 Follow-Up Period and Data Source for Each Site, by Outcome 
Final Report on the Performance-Based Scholarship Demonstration 



ARIZONA 

CALIFORNIA 

FLORIDA 

NEW MEXICO 

NEW YORK 

OHIO 

Outcome 

Credits earned/ 
attempted 

3 years 

Not available 

3 years 

4 years 3 

4 years 5 

4 years 

Enrollment 

3 years 

4 years 

3 years 

5 years 

5 years 3 

5 years 3 

Degrees earned 

3 years 

4 years 

3 years 

5 years 

5 years 3 

5 years 3 

Primary data source 

Pima 

Community 

College 

National Student 
Clearinghouse 

FHillsborough 

Community 

College 

University of 
New Mexico 

City 

University 
of New York 
(CUNY) d 

Ohio Board 
of Regents® 


NOTES: For all sites except California, National Student Clearinghouse data supplemented the data sets for enrollment and 
degrees earned. 

a Five years of data were processed for this outcome, but only 4 years were used since no other site had more than 4 years 
of follow-up data available. 

b Transcript and degree outcomes were missing for 33 students (out of 1,502) in the New York data provided by CUNY. 
These students were excluded from the credit value outcome impact analyses. The degree and enrollment outcomes for 
these students are based only on National Student Clearinghouse data for all 5 years of follow-up. 

c Year 5 includes only National Student Clearinghouse data. 

d lncludes data for all CUNY colleges. 

includes data from all Ohio public colleges. 


IMPLEMENTATION FINDINGS 

Evaluation studies of the individual PBS programs consistently found that the programs were suc- 
cessfully implemented in diverse settings . 25 While there was some variation by site, each program 
integrated the scholarship into the college (with the exception of California, where the program was 
not housed at a particular college), sometimes with the addition of supplemental service components. 


25. This section provides an overview of previous implementation findings. For more information, see individual 
reports published on each state: Patel and Rudd (2012); Patel and Valenzuela (2013); Sommo et al. (2014); Binder, 
Krause, Miller, and Cerna (2015); Mayer, Patel, and Gutierrez (2015); Richburg-Flayes et al. (2015). 
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Structure and Staffing 

At each institution, the program was supported by leadership at the college, who helped champion 
the program and customize it for the college and population it was meant to serve. Sometimes the 
program was housed in the financial aid department, other times in the student services depart- 
ment. In all cases, the program required a considerable amount of coordination between different 
groups. For example, the program at Pima Community College in Arizona involved a scholarship 
component combined with a comprehensive set of student services. In order to implement the pro- 
gram successfully, the program coordinator worked with the Learning Centers, student services 
staff, financial aid department, information technology (IT) department, and student accounts 
department. 

At each site, additional staff were hired to implement the program. Sometimes just one additional staff 
member was hired, but the number ranged up to three staff members for more complex programs. 
Consequently, at least one dedicated staff member was available to answer questions for students 
regarding the performance-based scholarship program. Each college had a program coordinator 
whose main responsibilities were to enroll students, check students’ eligibility requirements for 
each payment, and ensure that students were paid correctly. While some of these processes were 
automated to varying degrees, some amount of coordination with different departments was still 
necessary to implement the program. Through MDRC field visits and checks of student records, 
researchers confirmed that erroneous payments were rare. 

In California, the preexisting Cash for College program was an initiative to help low-income students 
apply for financial aid; it provided workshops on the application as well as one $1,000 scholarship 
to one or two randomly selected students at each Cash for College workshop with 25 attendees or 
more (as an incentive to increase attendance at the workshop). The program was adapted in 2008 
to implement performance-based scholarships, showing that a large-scale, portable scholarship 
program can be executed successfully. The study required more procedures in order to manage the 
various scholarship types, the increased volume of students in the program group, and the additional 
requirements attached to the disbursement of the award, as well as the portable nature of the schol- 
arships. Additionally, the study in California was built into a statewide program — a public-private 
partnership, bringing together high schools, community agencies, and businesses with a number of 
partner organizations. These organizations were able to adapt the performance-based scholarship 
design into their regular operations over the course of this study. 


Sites with Student Services Components 

Three of the six programs — those at Pima Community College in Arizona, Hillsborough Community 
College in Florida, and the University of New Mexico — successfully complemented the scholarship 
by requiring students to participate in support services in order to receive the award. Across these 
three states, service use increased markedly as a result of the programs. 

• At Pima Community College, students in the program group participated at high rates in 
the advising, tutoring, and workshops that were offered. For example, in the first term, over 
96 percent of students in the program group attended the orientation, nearly 84 percent met 
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with their advisers at least twice during the term, and almost 80 percent attended at least one 
Platica session. 26 

• At Hillsborough Community College, students in the program group were much more likely 
than students in the control group to visit the tutoring center. For example, in the first term, 
almost 89 percent of students in the program group visited the tutoring center, compared with 
60 percent of control group members. Students in the program group averaged 11.5 visits in the 
first term, compared with 4.9 visits from students in the control group. 27 

• At the University of New Mexico, students in the program group reported more visits (albeit 
shorter ones) with their adviser. Students in the program group saw an adviser on average about 
five times, while students in the control group did so about three times. The average length of the 
visit was about 15 minutes for students in the program group versus 18.5 minutes for students in 
the control group. Students in the program group appeared to have experienced greater satisfac- 
tion with the advisement process than students in the control group. 28 

Overall, researchers found that performance-based scholarships could be paired successfully with 
student services and used to promote their use. Most of the time, the services were ones that were 
already offered at the college — including tutoring and workshops at Pima Community College, 
tutoring at Hillsborough Community College, and advising at the University of New Mexico. This 
finding suggests that other colleges could also see an increase in the use of services at their campuses 
if a scholarship were tied directly to it. 


Scholarship Receipt 

Students generally responded positively to the scholarship offer and took advantage of the oppor- 
tunity. In each program, a majority of students earned at least one scholarship payment. Figure 1 
shows the total amount of scholarship dollars that students received (solid bars), compared with the 
amount available to them (dashed bars). 

In Arizona, where students had the opportunity to earn a part-time or full-time award, over the three 
semesters of the scholarship program students earned an average of $1,981 (around 44 percent) of the 
total of $4,500 available to them in the same period. Similarly, in Ohio, where there was a part-time 
and full-time differential, students earned $764 of the $1,800 available to them (about 43 percent). 
In New York, students were required only to earn a “C” or better in six credits or more to receive the 
full award; they received the largest proportion of the award available to them (77 percent). 


26. Patel and Valenzuela (2013). Some of the services in Arizona, such as tutoring and workshops, were also available 
to the control group. However, the advising that was offered to students in the program group in Arizona was 
different from that available to students in the control group. 

27. Sommo et al. (2014). 

28. Binder, Krause, Miller, and Cerna (2015). Calculations are from an online survey of second-cohort study participants 
conducted by the University of New Mexico. 
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FIGURE 1 Average Scholarship Amount Received Among Program 
Group Members, by Site: All Program Terms 

Final Report on the Performance-Based Scholarship Demonstration 
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Sites 


I Amount received 


Li Maximum amount for which students were eligible during first program year 


SOURCES: MDRC calculations using scholarship payment data from Pima Community College, the California Student Aid 
Commission, Hillsborough Community College, Lorain County Community College, Owens Community College, Sinclair 
Community College, the University of New Mexico, Borough of Manhattan Community College, and Hostos Community 
College. 

NOTE: Estimates differ from those in Table 3 because they are not adjusted by site and research cohort. 
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As the scholarship requirements grew more stringent, fewer students earned the scholarships, as 
might be expected. In the New York colleges, for example, where the benchmark was relatively low 
(getting a “C” or better in six credits or more), students earned the highest proportion of the schol- 
arship. At Pima Community College and the University of New Mexico, on the other hand, where 
the benchmarks were more stringent, students earned a lower proportion of the award. 


“IT MAKES ME FEEL THAT I CAN GO TO SCHOOL. 
IT TOOK AWAY STRESS. I WAS WORRIED I WOULD BE 
DROPPED FROM MY CLASSES BECAUSE YOU HAVE TO 
PAY YOUR FEES BY THE DEADLINE. I WAS ABLE TO USE 
THE SCHOLARSHIP TO PAY MY TUITION.” 

—A STUDENT IN CALIFORNIA 


On surveys, students reported that they primarily used the additional scholarship dollars for college- 
related expenses (like tuition, fees, or books) or basic living expenses. Students rarely reported using 
scholarship dollars in order to work fewer hours or to cover child care costs — even though a large 
number of students were working while attending college or were parents. This finding may be an 
indication that the timing of the scholarship was not appropriate or the amount was not enough for 
such students, given that the scholarships were intended to only partially augment their financial aid. 
It could be a sign that despite most students receiving financial aid, they still have school expenses 
and living expenses that are not fully covered . 29 


Lack of Unintended Consequences 

With any incentive program, there is a concern that students could respond to the incentives in ways 
that reasonably reflect the incentive structure but do not necessarily meet desirable outcomes from 
a policy perspective. For performance-based scholarships, these unintended consequences include 
the following possibilities: taking fewer credits in an effort to meet the benchmark (but potentially 
delaying graduation), selecting easier courses, reducing efforts at school (resulting in lower grades 
that cluster around the benchmark among students in the program group), and getting around 
the system in other ways (such as cheating, petitioning for grade changes, delaying developmental 
education courses, and so forth ). 30 


29. The surveys also examined other results, such as student motivation, time usage, level of effort toward studies, and 
employment patterns, as well as state-specific questions. For more information on the findings of these individual 
surveys, refer to Richburg-Hayes et al. (2015) and Sommo et al. (2014). See, also, Barrow and Rouse (2013). 

30. The literature on financial incentives is somewhat mixed on unintended consequences — many studies have noted 
the ones mentioned here. Cornwell, Lee, and Mustard (2005) report reduced full-time enrollment, increased course 
withdrawals, and concentration among "easier" majors in response to Georgia's HOPE program. There is a large 
literature in psychology that suggests external rewards can reduce intrinsic motivation such that once the rewards 
are removed, productivity declines. See Eisenberger and Cameron (1996) for a meta-analytic review that suggests 
the limited conditions under which these negative outcomes occur. 
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While it is not possible to effectively rule out all of these scenarios, the survey data captured across 
some of the sites provide some insight, namely in California, Florida, Arizona, and Ohio. Very few 
students reported taking easier or fewer classes in order to receive the award. Additionally, the survey 
examined motivation levels using the Relative Autonomy Index, which measures overall motivation 
toward a particular action (such as completing course work). 31 Across the states, students in the 
program group were not less motivated as a result of the award offer. In some cases, students in the 
program group exhibited an overall increase in motivation, albeit by a small amount. 


IMPACTS ON FINANCIAL AID AND STUDENT OUTCOMES 

Each of the programs in the Performance-Based Scholarship Demonstration was evaluated using 
a randomized controlled trial. Financial aid and academic data were used to estimate the impacts 
of the program on student outcomes. The durations of the programs varied, so the financial aid 
outcomes analyzed in this report were measured at the end of each program. For the same reason, 
the financial aid impacts are estimated separately for each program. 

A primary research question for this report is whether performance-based scholarships had an im- 
pact on long-term outcomes for enrollment, credits earned, and degree and certificate attainment. 
Consequently, the measures for academic outcomes use up to five years of follow-up data and are 
pooled across programs. Since the programs varied, pooling was not an obvious choice. California, 
for example, included scholarships of varying amounts and durations. It was also the only statewide 
program and the only program to target high school students before they enrolled in college, whereas 
other programs focused on students who had already enrolled at specific colleges, often including 
older students. The programs varied within a common model, however, so pooling allows for more 
precise estimates of the average long-term impacts of the model, as it was implemented in the dem- 
onstration. As described below, moreover, additional tests show little evidence that the impacts of 
the programs varied across the colleges. 

The analyses below show that the programs increased students’ financial aid over the duration 
of each program. These scholarships were intended to provide about 15 to 25 percent of students’ 
total aid during the programs. This amount is relatively modest in the context of students’ overall 
financial aid packages, particularly when the years following the program are considered. However, 
it was an amount that could nonetheless have some effect on student outcomes. The analyses also 
provide encouraging findings for the programs’ short- and long-term impact on students’ academic 
progress: The programs modestly increased enrollment, credit accumulation, and degree receipt. 

Box 1 explains how to read and interpret the impact tables that are discussed next. 


31. Deci, Koestner, and Ryan (2001). 
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BOX 1 




How to Read the Impact Tables in This Report 

Most tables in this report use the format illustrated in the table excerpt below, which displays cumulative 
credits earned by students in the program and control groups after Year 4. The row of data shows that 
students in the program group earned an average of 47.2 credits, and students in the control group 
earned an average of 45.1 credits. 

The "Impact Estimate" column in the table shows the observed difference in outcomes between the 
two research groups — that is, the estimated average impact of the opportunity to participate in the 
program. For example, the estimated average impact on credits earned can be calculated by subtracting 
45.1 from 47.2, yielding an impact estimate of 2.1 credits. 

Differences marked with one asterisk or more are considered statistically significant, meaning that there 
is a low probability that the difference occurred by chance. Differences that have no asterisk indicate 
that the opportunity to participate in the program did not have a discernible effect on that outcome. 
Assuming the true effect is zero, the number of asterisks indicates the probability that an estimate at 
least as large as the observed difference could have occurred by chance. One asterisk corresponds to a 
probability greater than 5 percent and less than or equal to 10 percent; two asterisks corresponds to a 
probability greater than 1 percent but less than or equal to 5 percent; and three asterisks correspond to 
a probability of 1 percent or less. The more asterisks that appear next to a positive difference, the more 
likely it is that the opportunity to participate in the program had a true positive average impact on the 
outcome. The p-value in the table identifies the probability. The impact in the table excerpt below has 
a p-value of 5 percent and two asterisks, indicating that the impact is statistically significant at the 5 
percent level — meaning that there is a 5 percent chance of observing an estimated average impact this 
large (or larger) if the opportunity to participate in the program actually had no average effect on credits 
earned. In other words, there is a 95 percent level of confidence that the opportunity to participate in 
the program had a positive impact on credits earned. 

Also shown in the table is the standard error of the impact estimate. The standard error is a measure of 
uncertainty or variability around the impact estimate. A useful guideline is that the confidence interval 
is usually calculated as 1 .96 multiplied by the standard error (for a 95 percent confidence interval). In the 
example below, the confidence interval is about 2.0 (1.96 multiplied by the standard error, or 1.96 x 1.04). 
Thus, there is a 95 percent chance that the "true" average impact on credits earned lies in the confidence 
interval between about 0.1 credit earned and 4.1 credits earned, calculated as 2.1 ± (1.96 x 1.04). 


CUMULATIVE 
CREDITS EARNED 


YEAR 

SAMPLE 

SIZE 

PROGRAM 

GROUP 

CONTROL 

GROUP 

IMPACT 

ESTIMATE 

STANDARD 

ERROR 

P-VALUE 

4 

4,835 

47.2 

45.1 

2.1 ** 

1.04 

0.05 


NOTE: Year 4, shown in this table excerpt, includes New Mexico, New York, and Ohio only. 
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Financial Aid 


Table 3 shows estimates of the impact on financial aid outcomes during each program. 32 The dura- 
tion of the financial aid outcomes in Table 3 varies by program length, during which students could 
increase their financial aid through performance-based scholarships. 

• The programs increased students’ financial aid. 

The first panel of Table 3 demonstrates that the PBS programs had a consistently positive impact 
on the total average financial aid that students received during the programs. The impact estimates 
are positive and statistically significant in each state, ranging from a low of $502 at the Ohio col- 
leges to a high of $2,209 at the New York colleges. The second panel of Table 3 shows that in each 
of the states, performance-based scholarship payments represented a substantial proportion of the 
financial aid increase. In each state, the estimates are positive and statistically significant, and they 
correspond closely to the overall estimated increases in financial aid. 


“[PBS] SOMEWHAT EMPOWERED ME BECAUSE I WAS ABLE 
TO GO AT THE PACE I NEEDED TO GO FINANCIALLY AND I 
WAS ALSO ABLE TO COVER THE COST ON MY OWN . . . 
AS AN ADULT STUDENT IT WAS JUST LIKE DECLARING 
AND TAKING OWNERSHIP . . . OF MY EDUCATION.” 

—A STUDENT IN NEW YORK 


• Three of the programs reduced students’ loans. 

The third panel of Table 3 provides evidence that three of the performance-based scholarship 
programs also helped students reduce their dependency on student loans. 33 At Pima Community 
College in Arizona and at the Ohio colleges, where students’ overall financial aid receipt was 
relatively low in the control groups ($6,602 and $7,445, respectively, as shown in the first panel), 
students in the program groups reduced their loans by an average of $352 and $334, respectively. 
At the University of New Mexico, where financial aid receipt was higher ($18,576 for students in 
the control group), students in the program group reduced their loans by an average of $606. In 
each of these states, students in the program groups took out smaller loans on average and had 
less student debt, but they still received more average financial aid in total when compared with 
students in the control group. 


32. Financial aid data were not collected for the California program. 

33. In some programs there were cases where loans had to be reduced in order for the performance-based scholarship 
to be awarded. In these cases, the colleges worked directly with students to ensure that they understood and were 
comfortable with their financial aid package. 
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TABLE 3 Financial Aid Outcomes, by Site: All Program Terms 
Final Report on the Performance-Based Scholarship Demonstration 


AMOUNT RECEIVED ($) 



SAMPLE 

SCHOLARSHIP 

PROGRAM 

CONTROL 

IMPACT 

OUTCOME 

SIZE 

DURATION 

GROUP 

GROUP 

ESTIMATE 3 

Total financial aid received 

Arizona 

1,028 

3 terms 

8,202 

6,602 

1,601 *** 

Florida 11 

1,075 

3 terms 

12,907 

11,372 

1,535*** 

New Mexico 

1,081 

4 terms 

20,487 

18,576 

1,911*** 

New York c 

1,502 

2 terms 

8,048 

5,839 

2,209*** 

Ohio 

2,285 

2 terms 

7,947 

7,445 

502*** 

Total performance-based scholarship received 

Arizona 

1,028 

3 terms 

1,977 

5 

1,972*** 

Florida 

1,075 

3 terms 

790 

0 

790*** 

New Mexico 

1,081 

4 terms 

2,597 

0 

2,597*** 

New York 

1,502 

2 terms 

2,002 

0 

2,002*** 

Ohio 

2,285 

2 terms 

765 

-1 

766*** 

Total loans received 

Arizona 

1,028 

3 terms 

1,218 

1,570 

-352** 

Florida 

1,075 

3 terms 

4,696 

4,564 

133 

New Mexico 

1,081 

4 terms 

2,415 

3,021 

-606** 

New York 

1,502 

2 terms 

1,042 

1,033 

10 

Ohio 

2,285 

2 terms 

2,853 

3,187 

-334*** 


SOURCES: MDRC calculations using financial aid data from Pima Community College, Hillsborough Community College, the 
University of New Mexico, the CUNY Institutional Research Database, Lorain County Community College, Owens Community 
College, and Sinclair Community College. 

NOTES: A two-tailed t-test was applied to differences between the research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; and * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

a The impact estimate is the estimated change in outcomes caused by the program, measured by the difference between 
the program and control group outcomes. 

b Financial aid data for Florida were available only for full academic years. The outcomes for Florida compare two full years 
of data. For the two cohorts randomly assigned in the fall, the data contain an extra term after the program ended. For the 
cohort randomly assigned in the spring, the data contain an extra term before the program began. 

The financial aid outcomes for the New York study represent financial aid awarded, which is distinct from financial aid 
received. The financial aid outcomes for the other studies shown represent financial aid received. 
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Academic Outcomes 


To estimate the programs’ average impacts on students’ academic outcomes over time, data are pooled 
across the programs, using as much follow-up data as possible from each program. Consequently, 
the analyses can detect much smaller impacts over a longer follow-up period than in previous re- 
ports. Each program is given equal weight for the pooled estimates in order to estimate the average 
impact of the programs, rather than weighing by students and giving studies with larger sample 
sizes more influence on the findings. Impacts on academic outcomes are estimated by comparing 
the weighted average outcomes for the students in the program and control groups. 34 Appendix A 
provides additional technical details about the analyses, as well as state-level impact estimates and 
other supplementary analyses. 

As discussed above, the follow-up data collected for each program range from three to five years 
following random assignment, so in the fourth and fifth years, outcomes from fewer programs 
are included in the pooled data. Consequently, the findings discussed here become somewhat less 
generalizable as the duration of the follow-up period grows longer. As shown below, however, the 
findings are fairly consistent across the programs and over time. 

“BEING PART OF [THE PROGRAM] WAS A STRESS-RELIEVER 

I I MADE ME FEEL BETTER ABOUT STUDYING, ABOUT 
GOING TO CLASS, AND ABOUT DOING THINGS OUTSIDE 

OF CLASS THAT I NEED TO DO.” 
— A STUDENT IN NEW MEXICO 


Finally, two of the programs included four-year colleges. In New Mexico, the PBS program was 
conducted at a four-year university, the University of New Mexico (see Box 2 for a description of the 
New Mexico program), and many students in the California study attended four-year colleges. For 
students who attended four-year colleges, evidence of impacts on outcomes, such as degree comple- 
tion, may take longer to materialize. 

• The programs increased enrollment by a small amount, on average. 

Table 4 presents pooled impact estimates on enrollment for each year of follow-up, measured by 
whether a student enrolled in college at any time during the given year. Enrolling and staying en- 
rolled in college is a key step toward graduation. The impact estimates in Table 4 suggest that, on 
average, the PBS programs had a small impact on enrollment, which was evident soon after the pro- 


34. The analyses reported here do not adjust the standard errors for clustering, because individuals were randomly 
assigned at every site in the demonstration and the program was administered to students at the individual level. 
Sensitivity analyses in which standard errors were adjusted for clustering at the cohort level on each campus were 
also performed. The sensitivity analyses lead to the same conclusions reported in this section. Results are reported 
in Appendix A. 
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BOX 2 

A Performance-Based Scholarship Program 
at a Four-Year Institution 

The performance-based scholarship program in New Mexico was the only program evaluated 
exclusively at a four-year institution, the University of New Mexico (UNM). The other programs in the PBS 
Demonstration operated at community colleges, with the exception of the California program, which 
operated at the state level and allowed students to take their scholarship to any accredited school, 
whether it was a two-year or a four-year institution. The structure and impacts of a performance-based 
scholarship program within the baccalaureate context are described below. 

The UNM program, called Vision Inspired Scholarship Through Academic Achievement (VISTA), 
targeted Pell-eligible first-year students, a very traditional college-going population* The goal of the 
UNM program was to create a scholarship that would provide incentives for students to stay on track 
to graduate with a bachelor's degree after four years. Specifically, the benchmark was for students to 
complete 12 credits in their first semester and 15 credits in their subsequent three semesters with a "C" 
grade or higher in each — the number of credits students would need in order to graduate in four years. 
The scholarship offered students up to $1,000 per semester for four semesters, or a total of $4,000 across 
two years. 

Students in the VISTA group were significantly more likely than students in the control group to attempt 
15 credits or more in the second semester, the minimum needed to earn the scholarship. The program 
was also estimated to increase the percentage of students who earned 30 credits or more by the end 
of their first year by 8.8 percentage points, improving the likelihood that they would be on track for an 
on-time graduation. The VISTA program also reduced students' loans by an estimated $600 over the 
four semesters during which students could earn scholarships — about 20 percent of the average loan 
amounts of students in the control group. 

There is also evidence that the VISTA program had an impact on degree receipt after five years: The 
estimated impact of the program is 4.4 percentage points (as shown in Appendix Table A. 5). Although 
this estimate is not statistically significant, it is close (p-value = 0.13), and when pooled with the two 
other sites for which five years of data are available, the pooled estimate of 3.3 percentage points is 
statistically significant (p-value = 0.02). The students at UNM were seeking bachelor's, not associate's, 
degrees, and no impacts on degree receipt were found in the first three years after random assignment 
— results that are not surprising since bachelor's degrees typically take at least four years to earn. 
However, these findings show that performance-based scholarships can improve student outcomes in 
four-year institutions as well as in two-year institutions, and the five-year results suggest that they can 
also have an impact on graduation rates for students seeking bachelor's degrees. 


* More information about the VISTA program is available in Miller, Binder, Harris, and Krause (2011) and Binder, 
Krause, Miller, and Cerna (2015). 
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TABLE 4 Impacts on Enrollment, Pooled 
Final Report on the Performance-Based Scholarship Demonstration 


PERCENTAGE ENROLLED 



SAMPLE 

PROGRAM 

CONTROL 

IMPACT 

STANDARD 


YEAR 3 

SIZE 

GROUP 

GROUP 

ESTIMATE 5 

ERROR 

P-VALUE 

1 

11,613 

95.6 

94.4 

1 2 *** 

0.32 

0.00 

2 

11,613 

79.3 

77.8 

1.5* 

0.77 

0.05 

3 

11,613 

67.8 

67.0 

0.8 

0.86 

0.38 

4 

9,510 

63.2 

61.7 

1.5 

0.96 

0.12 

5 

4,868 

42.9 

41.5 

1.4 

1.38 

0.30 


SOURCES: MDRC calculations using transcript data from the National Student Clearinghouse, Pima 
Community College, Hillsborough Community College, the University of New Mexico, the City University 
of New York, and the Ohio Board of Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Enrollment measures whether a student enrolled at any point in a given year. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

Estimates are weighted to account for the different sample sizes and random assignment ratios in each 
state. 

a Years 1, 2, and 3 each include Arizona, California, Florida, New Mexico, New York, and Ohio. Year 4 
includes California, New Mexico, New York, and Ohio. Year 5 includes New Mexico, New York, and Ohio. 

b The impact estimate is the estimated change in outcomes caused by the program, measured by the 
difference between the program and control group outcomes. 


grams started. 35 In Year 1, for example, an estimated 95.6 percent of students in the program group 
enrolled in college, compared with 94.4 percent of students in the control group, for a statistically 
significant impact estimate of 1.2 percentage points. In each subsequent year, the estimated impact 
on enrollment is positive. In Year 2, the estimated impact of 1.5 percentage points is statistically 
significant and constitutes about a 2 percent gain over the control group enrollment rate of about 
78 percent. In Year 4, the estimate is also 1.5 percentage points, but it is not statistically significant 
at the 0.1 level (with a p-value of 0.12). 

Although the performance-based scholarship programs in the PBS Demonstration appear to be help- 
ing a small number of additional students enroll in college, these estimates are considerably smaller 
than in the evaluation of the Opening Doors Louisiana performance-based scholarship program. 
That program increased enrollment in the second semester by an estimated 15 percentage points 


35. The impact in Year 1 was primarily due to the California program (see following), which was not the case in 
subsequent years. 
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over the control group enrollment rate of about 50 percent, and by an estimated 12 percentage points 
over the control group rate of about 38 percent in the third semester. 36 However, long-term follow-up 
was disrupted. It is not clear why the gains in Louisiana were larger, but they may have been due to a 
number of factors, including differences in student characteristics; economic conditions, including 
low unemployment rates; and lower overall enrollment rates (with only about half of the students 
in the control group enrolled in the second semester). 

• The results from the study in California suggest that targeting high school seniors can have 
an impact on first-year enrollment. 

The California program was the only program to target high school seniors, and it drove most of 
the pooled impact estimate in the first year: Only the California program produced a statistically 
significant impact estimate on first-year enrollment — about 4 percentage points over the control 
group enrollment rate of about 89 percent. The impacts on first-year enrollment in California were 
also concentrated among students who attended community colleges. Community colleges generally 
allow students to make later decisions about enrollment than four-year institutions do. This differ- 
ence may have been an important factor that contributed to higher enrollment rates at community 
colleges but not four-year institutions, since the California students were notified of their scholarship 
eligibility in June. 37 The other PBS programs were unlikely to affect first-year enrollment since they 
targeted students who had already enrolled in college. Appendix Table A. 2 provides the state-level 
impact estimates for each year of the study. The table also provides evidence that the impact on 
first-year enrollment in California differed from that in the other states. 

• The programs produced modest, positive impacts on credit accumulation. 

Despite the relatively small impact estimates on enrollment, previous studies of the PBS Demonstration 
programs reported encouraging results in terms of meeting academic benchmarks, outcomes on which 
the performance-based scholarship programs were most likely to have an impact in the short-term. 
The academic benchmarks that students needed to meet in order to earn the scholarships typically 
consisted of earning a “C” or better in a minimum number of credits. Students in almost all of the 
program groups were more likely than those in the control groups to meet their scholarships’ end- 
of-term benchmarks in at least one term. 38 Consequently, the programs may also have had an effect 
on longer-term outcomes like overall credit accumulation. 

Table 5 presents pooled impact estimates on cumulative credits earned by the end of each year 
of follow-up, measured by the total number of credits students earned each year, for both devel- 
opmental and college-level courses. The estimates in Table 5 suggest that PBS had a consistently 
positive impact on credit accumulation. In each year of follow-up, the impact estimates are positive 


36. Richburg-Hayes et al. (2009). The second semester outcomes are for all cohorts at Delgado Community College and 
Louisiana Technical College. The third semester outcomes are for the first two cohorts only at Delgado Community 
College and Louisiana Technical College. 

37. Richburg-Hayes etal. (2015). 

38. The program in New York was the single exception. See the appendix to Patel, Richburg-Hayes, de la Campa, and 
Rudd (2013) for estimated impacts on end-of-term benchmarks. 
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TABLE 5 Impacts on Cumulative Credits Earned, Pooled 
Final Report on the Performance-Based Scholarship Demonstration 


CUMULATIVE 
CREDITS EARNED 



SAMPLE 

PROGRAM 

CONTROL 

IMPACT 

STANDARD 


YEAR 3 

SIZE 

GROUP 

GROUP 

ESTIMATE 13 

ERROR 

P-VALUE 

1 

6,938 

17.5 

16.2 

1 2 *** 

0.24 

0.00 

2 

6,938 

28.6 

26.7 

-| g*** 

0.46 

0.00 

3 

6,938 

36.4 

34.6 

1.8*** 

0.64 

0.00 

4 

4,835 

47.2 

45.1 

2.1** 

1.04 

0.05 


SOURCES: MDRC calculations using transcript data from Pima Community College, Hillsborough 
Community College, the University of New Mexico, the City University of New York, and the Ohio Board of 
Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

Estimates are weighted to account for the different sample sizes and random assignment ratios in each 
state. 

a Years 1, 2, and 3 each include Arizona, Florida, New Mexico, New York, and Ohio. Year 4 includes New 
Mexico, New York, and Ohio. 

b The impact estimate is the estimated change in outcomes caused by the program, measured by the 
difference between the program and control group outcomes. 


and statistically significant. During Year 1, for example, when all students in the program groups 
could receive scholarships from the PBS program, they earned an estimated 17.5 credits on average, 
compared with 16.2 credits for students in the control group, for a statistically significant impact 
estimate of 1.2 credits — a 7.4 percent gain over the control group. This positive short-term finding 
coincides with the encouraging findings for each program for the end-of-term benchmarks. By the 
end of Year 4, at least two years after each PBS program had ended, the estimated impact on credits 
earned increased to 2.1 credits — a 4.7 percent gain. 

Earning credits is a key marker of students’ progress toward a degree, and the estimates in Table 
5 show that the PBS programs supported students in this regard. Many classes are worth about 3 
credits, so a 2.1 credit impact suggests that, on average, the performance-based scholarship helped 
many students attempt and pass just under one additional course than they would have without 
the program. (See Appendix Table A. 3 for state-level estimates for credits earned.) It is also notable 
that the estimated impacts for credits earned correspond closely to the estimated impacts on credits 
attempted: The estimated impacts on credits earned are generally 0.2 or 0.3 credits larger than the 
estimates for credits attempted. (See Appendix Table A. 4 for estimates on credits attempted.) The 
PBS programs appear to have encouraged students to take and earn additional credits. However, 
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the estimated impacts on credits earned are larger than those on credits attempted in every year, 
suggesting that the programs also helped students pass a higher proportion of their classes. 

• The programs produced modest, positive impacts on degree attainment. 

Table 6 presents pooled impact estimates on the percentage of students who earned a degree for 
each year of follow-up, measured by whether students earned a certificate, an associate’s degree, or 
a four-year degree at any college. Earning a credential is a critical outcome for students and a key 
goal for college programs because it improves students’ employment and earning prospects. Table 6 
shows, however, that by the end of Year 4, only about 26 percent of students in the control group had 
earned a two- or four-year degree. By the end of Year 5, the percentage rises to just about 35 percent. 

Table 6 suggests that the PBS programs had a modest impact on the proportion of students earning 
a degree and helped some students graduate. By the end of Year 2, for example, an estimated 8.3 
percent of students in the program group had earned a degree, compared with 7.4 percent of students 
in the control group, a statistically significant impact estimate of 0.9 percentage points — or a 12.1 
percent gain over the control group. The impact estimates are positive in each year of follow-up and 
are statistically significant in Year 2, Year 3, and Year 5. By the end of Year 5, the impact estimate 
grows to 3.3 percentage points — a 9.4 percent gain over the control group completion rate of 35.1 
percent. Recall, however, that the sample analyzed changes in later years of follow-up. While students 
from all programs are included in the Year 1, Year 2, and Year 3 estimates, students from the studies 
at Pima Community College in Arizona and Hillsborough Community College in Florida are not 
included after Year 3, and students in the California study are not included after Year 4. Notably, the 
impact estimates for degree receipt in the Florida program are positive and statistically significant 
(at the 0.1 level) in both Year 2 and Year 3 (as shown in Appendix Table A. 5), despite the relatively 
modest findings for other short-term outcomes. 39 Although the pooled findings in Year 5 apply to 
fewer colleges, they nonetheless suggest that the programs positively affected degree receipt. 

• There is little evidence that impacts differed across programs. 

For each outcome — enrollment, credits earned, and degrees earned — and for each year of follow- 
up, statistical tests were used to examine whether state-level impact estimates were different. 40 Only 
2 out of these 14 tests suggest that the impact estimates differed at statistically significant levels 
across states. 41 


39. Sommo et al. (2014). 

40. A joint F-test was used to assess impact variation among the states. This test measures whether any predictive 
power is gained by including indicators in the model that identify participants' state and treatment condition, 
compared with just including a treatment indicator. Appendix A provides impact estimates and additional details 
about the F-test. 

41. Two tests produced p-values below the 0.1 level. As discussed above, one of these tests suggests that California is 
driving the Year 1 enrollment estimates (p-value = 0.01). The other test for statistically significant differences is for 
degree outcomes in Year 2. The programs in Ohio and Florida were the only programs that produced statistically 
significant impact estimates on degree outcomes in Year 2, and similarly for Year 3. It is possible that those 
programs are driving the shorter-term degree impact estimates. (The test for differences produces a p-value of 0.06 
in Year 2 and a p-value of 0.17 in Year 3. See Appendix Table A. 5.) 
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TABLE 6 Impacts on Degrees Earned, Pooled 
Final Report on the Performance-Based Scholarship Demonstration 


PERCENTAGE WHO 
EARNED A DEGREE 



SAMPLE 

PROGRAM 

CONTROL 

IMPACT 

STANDARD 


YEAR 3 

SIZE 

GROUP 

GROUP 

ESTIMATE 13 

ERROR 

P-VALUE 

1 

11,613 

2.9 

2.8 

0.2 

0.3 

0.61 

2 

11,613 

8.3 

7.4 

0.9* 

0.5 

0.07 

3 

11,613 

14.9 

13.6 

1.4** 

0.7 

0.04 

4 

9,510 

26.4 

25.7 

0.7 

0.9 

0.44 

5 

4,868 

38.3 

35.1 

3.3** 

1.4 

0.02 


SOURCES: MDRC calculations using transcript data from the National Student Clearinghouse, Pima 
Community College, Hillsborough Community College, the University of New Mexico, the City University 
of New York, and the Ohio Board of Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

Estimates are weighted to account for the different sample sizes and random assignment ratios in each 
state. 

a Years 1, 2, and 3 each include Arizona, California, Florida, New Mexico, New York, and Ohio. Year 4 
includes California, New Mexico, New York, and Ohio. Year 5 includes New Mexico, New York, and Ohio. 

b The impact estimate is the estimated change in outcomes caused by the program, measured by the 
difference between the program and control group outcomes. 


• There is little evidence that the impacts varied with program characteristics, but the demonstra- 
tion was not designed to determine the relative effectiveness of different program characteristics. 

The programs in Arizona, Florida, and New Mexico included services in addition to the scholarship, 
whereas programs in other states did not . 42 Impact estimates for programs that provided services 
were compared with the estimates for programs that provided only scholarships. These analyses do 
not provide evidence that the impacts were different for programs that required students to use ad- 
ditional support services, when compared with programs that did not include such requirements . 43 
The demonstration, however, was not designed to determine whether including services made the 
programs more effective. Other factors — including characteristics of the student populations and 


42. In Louisiana, not included in the analyses for this report, the services were more limited and involved only meeting 
with an adviser. 

43. The only significant difference is in enrollment in the first year, but again, this may be driven by the design of the 
scholarship programs in California. 
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contextual features at the colleges themselves — varied as well, and cannot be untangled from the 
effects of services. It may be that the additional services did not improve student outcomes, but it is 
also possible that the services did help students to do better than they would have otherwise. When 
program designers included these services, moreover, they did so because they believed the services 
would help students overcome barriers that inhibited academic progress. Appendix Table A. 6 shows 
the full results of the analyses. 


“I PROBABLY WOULDN’T HAVE KNOWN ABOUT THE 
MATH LAB IF NOT FOR MAPS [HILLSBOROUGH’S PBS 
PROGRAM]. . . . BEFORE THIS PROGRAM, I TOOK . . . THREE 
YEARS OFF, AND I TRIED AN ONLINE COURSE AND I WAS 
ON MY OWN WITHOUT THE MATH LAB.” 

—A STUDENT IN FLORIDA 


In California, the evaluation was designed to test whether scholarships that varied by duration and 
amount produced different impacts. Five different performance-based scholarship types were tested 
against a control condition in which students did not receive a scholarship and against a $1,000 
scholarship in one semester that was not performance-based. The performance-based scholarships 
ranged from a potential maximum award of $1,000 to $4,000 and lasted between one semester and 
four semesters. Analyses of these scholarship types also provide little evidence that the different 
scholarships produced different impacts, but the analyses had limitations. For one, students could 
take the scholarship to any college, but for a large portion of the sample, transcript data were not 
available. Consequently, evidence of different impacts on credit accumulation — the outcome where 
PBS programs have shown the most consistent evidence of effects — could not be examined for the 
full sample. The California study was also designed to detect differences between scholarship types 
that are larger than the pooled impact estimates for enrollment and graduation reported above. To 
detect differences due to duration, for example, scholarships that last for two years would need to 
produce a gain of at least 6 percentage points over scholarships that last for one year or less. Increasing 
the sample size to detect smaller impacts, however, was cost-prohibitive. Consequently, it is possible 
that differences in duration and amount could produce meaningful differences in impacts. 44 

Appendix Tables A. 7 (enrollment) and A. 8 (degrees earned) show impact estimates for each scholar- 
ship type and the p-values for the F-tests for each year. 

• The impacts do not appear to differ across different subgroups of students. 

In each year of follow-up, for each of the three outcomes, statistical tests examined whether impacts 
were different for student subgroups defined by gender, whether the students were Latino, whether 
the students were parents, whether the students were younger than 20 years old, whether they were 
the first in their family to attend college, and whether they were employed at the beginning of the 


44. For more information about the California program and evaluation, see Richburg-Hayes et al. (2015). 
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program. These analyses provide little evidence of variation in impacts and suggest that the pro- 
grams were similarly effective across different groups of students. 45 Notably, the estimated impacts 
on credits earned are positive and statistically significant for all student subgroups in each of the 
first two years. Appendix Tables A. 9 (enrollment), A. 10 (credits earned), and A. 11 (degrees earned) 
provide more detail. 

In many cases, particular groups of students were expressly targeted by the programs. Consequently, 
these subgroups are correlated with the programs themselves, so the relationships between subgroups 
and outcomes are difficult to untangle from other factors that vary, such as context and program 
characteristics. 


Summary 

These findings suggest that the performance-based scholarship programs generally worked as de- 
signed. The analyses also detected little variation in the impact estimates, although the evaluation 
was not designed to determine whether some program configurations were more effective than others. 
Instead, program designers had flexibility to create scholarship programs to meet the needs of their 
student populations. It is possible that programs that included services may have been more effective 
because of those services. The implementation data discussed previously, for example, suggest that 
students used additional services when they were included as program components. 

The programs’ scholarships covered a relatively small proportion of students’ full cost of college 
during the program, and even less when considering the full follow-up period during which stu- 
dents continued to attend college. However, the analyses presented here provide evidence that the 
performance-based scholarships helped students make greater academic progress toward their de- 
grees and modestly improved degree receipt. This impact appears to have occurred because the PBS 
programs encouraged students to take additional classes, but also because students passed more of 
the classes they took. The scholarships provided financial support, and in some cases the programs 
required students to use additional services that were intended to provide academic support, suc- 
cessfully increasing students’ participation in those services. In three cases, the programs reduced 
students’ debt, a growing issue in national conversations about the costs of college. Notably, the 
analyses described here also suggest that the performance-based scholarship programs were similarly 
effective across the states and for different student subgroups. 


COSTS AND COST-EFFECTIVENESS 

Financial data were used to estimate the cost of the programs in the Performance-Based Scholarship 
Demonstration and to examine how the programs affected the average total cost of each degree that 
students earned. The average direct cost of a performance-based scholarship program was $2,345 


45. A small number of differences between subgroups were statistically significant — about the same number that 
would be expected to occur by chance. The differences also were not consistent across outcomes. Less than 4 
percent of subgroup comparisons were statistically significant at the 5 percent level, and about 8 percent were 
statistically significant at the 10 percent level. 
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per student over the life of the scholarship offer. Most of this investment (71.5 percent) was associ- 
ated with payments to students, while about one-fourth (26.1 percent) was associated with program 
administration. After five years of follow-up, the cost per degree was nearly equal for students in 
the program group and the control group. (The average cost per degree was about 0.8 percent less 
for the program group.) The increase in the proportion of program group members earning a degree 
(an average increase of 9.3 percent over the control group) was proportionally similar to the group’s 
increase in costs (an average increase of 8.4 percent over the control group). 


Methodology 

All costs have been adjusted to 2014 dollars for this analysis. 46 The analysis aims to illustrate the 
cost of steady-state operation for an average college that offered performance-based scholarships. 
As a result, this analysis excludes start-up and evaluation costs. 47 Costs are estimated using college 
financial data. Since all funds (such as tuition paid by students; subsidies from the local, state, and 
federal governments; and private donations) are funneled through the college, this approach provides 
a good estimate of the total investment made in these students. 

The direct cost of the performance-based scholarship programs is estimated using program ex- 
penditure reports and informed by staff interviews to ensure that all of the resources required to 
operate the program were accounted for in the analysis. The cost of college services is estimated for 
students in the program and the control groups separately. The cost of college services is based on 
the average cost per credit attempted, multiplied by the number of credits attempted by the respec- 
tive group. 48 This analysis presumes that additional credits attempted by students in the program 
group, above and beyond the level attempted by students in the control group, have zero marginal 
cost. 49 Finally, in order to calculate cost-per-graduate values, the direct cost and the cost of college 


46. This analysis used the Higher Education Price Index (HEPI) for all public two-year colleges to adjust all costs to 
constant 2014 dollars. 

47. If start-up and evaluation costs were included, the total cost estimate would increase by 4.9 percent ($115 per 
sample member). If start-up costs were included but evaluation costs excluded, the total cost estimate would 
increase by less than 1 percent ($17 per program group member). 

48. The average cost per credit attempted is estimated for the individual institutions by dividing the college's annual 
expenditures by the total number of credits attempted by students at the college during the year of interest, as 
reported in the Integrated Postsecondary Education Data System (IPEDS). For credits attempted throughout the 
follow-up period, an average cost per credit across sites is calculated using data from Pima Community College 
in Arizona; Hillsborough Community College in Florida; the University of New Mexico; Borough of Manhattan 
Community College and Hostos Community College in New York; and Lorain County Community College, Owens 
Community College, and Sinclair Community College in Ohio. Data on credits attempted were not available for the 
entire sample in California. The notes to Appendix Table A. 12 include more detail about how the number of credits 
attempted was estimated. 

49. The base case assumes zero marginal cost, which means the college is presumed to be able to fully absorb the cost 
by leveraging existing resources — for instance, by having students in the program group attempt more credits 
by taking courses where seats were previously empty. If a college does not absorb the cost of additional credits 
attempted by students in the program group, then the cost of the college experience for students in the program 
group will be higher. As a result, a sensitivity adjustment is included in this analysis to illustrate how results would 
change if the analysis assigned a cost to additional credits attempted by program group members beyond the 
number of credits attempted in the control group. 
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services are added for each group, and then divided by the total number of graduates in that group. 
This calculation yields a cost-per-graduate value for students in both the program and control 
groups. Comparing these values shows whether the PBS program was more efficient or less efficient 
at producing graduates in comparison with the usual college services. 


Direct Cost of Performance-Based Scholarships 

The direct cost of performance-based scholarship programs includes the cost of scholarship payments, 
administration, and support services. Figure 2 shows the cost per PBS program group member. The 
height of each bar in Figure 2 shows the full per-student average cost. Each bar is also disaggregated 
into categories: scholarship payments, administration, and student services. The first bar in Figure 
2 shows that the average cost of a performance-based scholarship program was $2,345 per student. 
As discussed earlier in this report, programs varied in their duration, amount offered, benchmarks 
required, and actual amount paid, all of which affect the direct cost. The average cost ranged from 
a per-student cost of $1,150 at the Ohio colleges (averaged across the three colleges) to $3,828 at the 
University of New Mexico. 50 Figure 2 also shows that scholarship payments accounted for the major- 
ity of the cost of implementing performance-based scholarship programs. On average, an estimated 
$1,678, or 71.5 percent of the total cost, was paid to students as performance-based scholarships. 
Scholarship costs varied across the sites; the Ohio colleges paid the average student $816 in scholar- 
ships, while the University of New Mexico paid $3,002. The percentage of total costs associated with 
scholarship payments ranged from 62.9 percent at Hillsborough Community College in Florida to 
78.4 percent at the University of New Mexico. 

The second largest component of program cost was administration. On average, colleges spent $612 
per student (26.1 percent of the total cost, shown in the first bar of Figure 2) on administration. 51 
Administrative costs ranged from $334 per student at the Ohio colleges to nearly one thousand 
dollars ($958) at the New York colleges. The percentage of program costs spent on administration 
ranged from 19.2 percent at the University of New Mexico to 29.8 percent at the New York colleges. 

The smallest expenditure category was student services. When all programs are included, the pro- 
grams spent an average of $55 per student for student support services (or 2.4 percent of the total 
cost). Among only the three programs that included additional student services (at Pima Community 
College in Arizona, Hillsborough Community College in Florida, and the University of New Mexico), 


50. In California, where there were five different scholarship types (with varied durations and offer amounts), the 
scholarship costs ranged from $903 to $2,518 across the scholarship types. 

51. In other higher education cost analyses, MDRC has observed that administrative costs for innovative programs 
operating at a fairly small scale (for instance, serving a fraction of students at a handful of schools) often account 
for around 20 percent of total program costs. For instance, in a multisite demonstration of learning communities, 

17 percent of costs were associated with administration (Visher et at, 2012). Similarly, an evaluation of the City 
University of New York's Accelerated Study in Associate Programs (ASAP) estimated the cost of administration 

as 23 percent of total costs (Scrivener et at, 2015). Performance-based scholarships have higher administrative 
costs because adding conditionality simultaneously lowers the amount of payments and increases the cost of 
administration, which makes administrative costs a larger fraction of the total program cost. If a program increased 
scholarship amounts or the number of participating students, or if the program were in operation longer, the share 
of total costs dedicated to administration would likely decrease. 
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FIGURE 2 Cost of PBS Program per Program Group Member 
Final Report on the Performance-Based Scholarship Demonstration 
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FIGURE 2 (continued) 


SOURCES: MDRC calculations based on each site's expenditure data for its performance-based scholarship program, 
financial aid and scholarship impact data, and the Higher Education Price Index. 

NOTES: All costs have been adjusted to constant 2014 dollars. 

Program costs are based on a steady state of operation that excludes external research and start-up costs. If 
start-up and evaluation costs were included, the total cost estimate would increase by 4.9 percent ($115 per sample 
member). If start-up costs were included but evaluation costs excluded, the total cost estimate would increase by 
less than 1 percent ($17 per sample member). 

California's analysis includes five different types of performance-based scholarships. A sixth scholarship type, 
which was not performance-based, has been excluded, as in the impact analysis. 

In New York, colleges offered scholarships of up to $2,600 per student to 377 students and offered PBS Plus 
Summer scholarships of up to $3,600 per student to 378 students. 

California, New York, and Ohio did not offer additional student services to students in PBS programs and as such 
did not incur any costs related to student services. 

Tests of statistical significance were not performed. 

Rounding may cause slight discrepancies in sums and differences. 


the average cost of student services was $111, or 4.0 percent of the total cost. Overall, the cost of 
student services appears small in comparison with the cost of administering payments. This small 
investment, however, leveraged existing student services (to provide students in the program group 
with more services than they otherwise would have received). 52 


Cost-Effectiveness Analysis 

Figure 3 illustrates the cost-effectiveness of the PBS program. In the first panel, the black portion of 
the bars shows the cumulative average cost of college services. This analysis assumes that the cost 
of college services for students in both the program group and the control group is the same, mean- 
ing that additional credits attempted are assumed to have zero marginal cost. Below, a sensitivity 
adjustment explores an alternate assumption, wherein the cost of college is not the same for both 
groups, and additional credits attempted do produce an additional cost. The white bars in the first 
panel show the average additional cost for the PBS program. The relative cost of the performance- 
based scholarships becomes smaller over time as students take additional classes, thus incurring 
larger costs for college services, which steadily account for a larger proportion of the total cost. 
For example, in Year 1, the cumulative average additional cost for the program group was $2,345 
(shown in Figure 2), a 25.6 percent increase over the $9,160 invested in the average control group 
member. Over time, the cost of the PBS program becomes proportionally smaller when compared 
with the cumulative average cost of college services for the control group. By Year 5, the cost of the 
PBS program represents an increase of only 8.4 percent. 


52. The average college in the study devoted 8.3 percent and 10.7 percent of total yearly expenses to academic support 
and student services, respectively (estimates based on data reported to IPEDS). 
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FIGURE 3 Cost-Effectiveness of PBS Program 
Final Report on the Performance-Based Scholarship Demonstration 


Total cumulative cost ($): The relative increase in investment from PBS decreases over time 
as the cost of college services increases. 
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FIGURE 3 (continued) 


SOURCES: MDRC calculations using transcript data from the National Student Clearinghouse, Pima Community 
College, Hillsborough Community College, the University of New Mexico, the City University of New York, and 
the Ohio Board of Regents, as well as program-specific budget data and financial and enrollment data from the 
Integrated Postsecondary Education Data System and the Higher Education Price Index. 

NOTES: Program costs are based on a steady state of operation that excludes external research and start-up costs; 
the cost would increase by 0.7 percent if start-up costs were included and by 4.0 percent if research costs were 
included. 

All costs were adjusted into constant 2014 dollars. 

Rounding may cause slight discrepancies in sums and differences. 

The cumulative cost of college services is estimated based on credits attempted. For additional detail about how 
the cumulative cost of college services is estimated by year, see Appendix Table A.12. 

a The cost per degree for both students in the program and students in the control group is calculated by adding 
the cost of usual college services to the direct cost of the average PBS program and dividing by the estimated effect 
on earning a degree. For example, to calculate the 5-year cost per degree for the average program student, the 
5-year cost of college services ($27,809) is added to the direct cost of the average PBS program ($2,345) for a total 
of $30,154 ($27,809 + $2,345 = $30,154). This total cost is then divided by the estimated percentage of program 
students who earned a degree after Year 5 (38.3 percent) for a cost per degree of $78,672 ($30,154/(38.3 / 100] = 
$78,672). Discrepancies in calculations are due to rounding of impact estimates and costs. 


The second panel of Figure 3 shows the percentage of students who earned a degree, by year. Over 
time, more students in the program and control groups graduated. The estimated program impact 
changes from year to year. The black portion of the bars in the figure illustrates the base graduation 
rate without the PBS program, or the graduation rate of the students in the control group. The white 
portion of the bars illustrates the estimated impact (the difference between the program group and the 
control group) on graduation due to the program. For example, after the first year, 2.8 percent of the 
control group had graduated, and the estimated impact due to the PBS program was 0.2 percentage 
points (not statistically significant at the 10 percent level), representing a 5.8 percent increase in the 
graduation rate. In Year 5, the final year of follow-up, 35.1 percent of students in the control group 
graduated, and the estimated impact due to the PBS program was 3.3 percentage points (statistically 
significant at the 5 percent level), representing a 9.3 percent increase in the graduation rate. 

The black bars in the third panel of Figure 3 show the cost per degree for students in the control group, 
by year. The cost per degree is large in the early years because few students earned a degree during that 
time frame. Over time, the cost per degree decreases as more students earn degrees. The white bars 
illustrate the estimated impact on cost per degree due to the PBS program. The labels over the black 
and white bars indicate the percentage change in the cost of earned degrees between students in the 
program group and the control group. Over time, the cost per degree for the two groups generally 
became more similar. In Year 1, the cost per degree in the control group was approximately $331,000, 
while it was 18.7 percent more expensive (approximately $393,000) for students in the program group. 
By Year 5, the cost per degree in the control group was $79,310, and for those receiving performance- 
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based scholarships it was 0.8 percent lower, or $78, 672. 53 The estimated impact on degrees earned 
was slightly larger, proportionately, than the estimated increase in investment associated with the 
scholarships. Specifically, the program improved graduation rates by an estimated 9.3 percent while 
it increased the level of investment in program students by an estimated 8.4 percent. Appendix Table 
A. 12 shows detailed values for the cost-effectiveness of performance-based scholarships. 


Sensitivity Adjustment 

The main analysis assumes that the marginal cost of additional credits attempted is zero. The sen- 
sitivity analysis described here assumes the marginal cost of an additional credit attempt is equal 
to the average full cost of a credit attempt. 54 

Under this assumption, if the colleges were unable to absorb the cost of the additional credits at- 
tempted by students in a PBS program, then after five years the colleges would have needed resources 
to cover the cost of each student offered a performance-based scholarship attempting about 1.4 more 
credits on average. The average cost of college services for these students would then be $28,347, 
compared with the average cost of $27,809 in the control group, an additional cost of $539. As a result, 
after five years, the cost per degree for students in the program group would increase to $80,077, 
approximately 1.0 percent higher than the control group cost of $79,310. Even under this assump- 
tion, the PBS programs would still have produced more graduates for a cost that is very similar to 
the cost of graduates in the control group. 


CONCLUSIONS 

The national Performance-Based Scholarship Demonstration provides evidence that performance- 
based scholarships modestly improve both short- and long-term academic outcomes. The analyses 
in this report use data from six randomized controlled trials of scholarship programs, including 
over 12,000 low-income students who attended two- and four-year institutions, to evaluate the ef- 
fectiveness of these scholarships for a diverse set of student populations. Overall, the analyses show 
the following: 


53. Research suggests that a cost of about $80,000 per associate's degree is well below the economic value of an 
associate's degree. For example, Levin and Garcia (2013) estimate the after-tax benefit of an associate's degree in 
New York City to be $160,000 to the student (from increased lifetime earnings) and $200,000 to taxpayers. Abel and 
Deitz (2014) estimate the benefit of an associate's degree to be $325,000, based on estimated increases in lifetime 
earnings. 

54. The cost per additional credit attempted may vary in other ways. For instance, it could be greater or less than the 
average cost per credit, depending on the college. Some colleges may have extra capacity to absorb the additional 
credits attempted. Other institutions may have less excess capacity and require more resources to absorb the 
additional credits attempted. After five years of follow-up, the average student in a performance-based scholarship 
program attempted a projected 71.2 credits, while students in the control group attempted a projected 69.9 credits, 
a difference of 1.4 credits attempted. For additional detail on how the credits attempted in Year 5 were projected, 
see the notes to Appendix Table A. 12. 
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• Performance-based scholarships can be implemented at the college level or state level, in order 
to meet the goals of individual institutions or intermediaries. Each of the sites in the six states 
across the demonstration integrated the scholarship into their existing systems. In California 
in particular, the program was implemented as part of a statewide effort to improve academic 
outcomes for college-going high school students. 

• In sites where student services accompanied the performance-based scholarship, service 
use increased markedly as a result of the program. At Pima Community College in Arizona, 
Hillsborough Community College in Florida, and the University of New Mexico, student services 
such as tutoring, advising, and workshops were integrated in various ways with the award. The use 
of these services increased as a result of the program incentives associated with the scholarship. 55 

• Offering performance-based scholarships to students increases their total financial aid, even 
when helping to reduce their dependency on loans. The PBS program consistently increased the 
amount of financial aid that students in the program group received, ranging from around $500 
at the Ohio colleges to $2,200 at the New York colleges. Additionally, in three of the six states, the 
program decreased students’ loans during the time that they were offered a scholarship, ranging 
from reductions of about $330 to $600. 

• The programs have a positive impact on short-term outcomes and produce positive, mod- 
est impacts on longer-term outcomes. Performance-based scholarships improved enrollment 
between 1 and 2 percentage points in the first two years of follow-up, on average. Almost all of 
the programs also increased the percentage of students meeting the end-of-term benchmarks — 
generally receiving a “C” or better in a minimum number of credits — in at least one term. 56 By 
the end of the fourth year, students in the program groups had earned an estimated average 2.1 
credits more than students in the control groups. 

• The programs modestly increased degree completion. After five years, the estimated impact on 
degree completion is 3.3 percentage points, over a completion rate of 35.1 percent in the control 
group. 

• While the scholarship program varied at each site (in terms of target population, amount 
awarded, duration, and other details), there is little evidence that the impacts from the pro- 
grams varied. This finding holds even when looking across a variety of dimensions: PBS programs 
appear to work in various scholarship programs, with varying scholarship designs, in different 
state and academic settings, and for diverse student populations, including younger and older 
students, men and women, and Hispanic and non-Hispanic students. 

• Evidence from the study in California suggests that offering students a scholarship in their 
senior year of high school increases enrollment in the first year of college. The program in 


55. Binder, Krause, Miller, and Cerna (2015). 

56. New York was the single exception. 
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California was the only site to offer scholarships to high school students. It was also the primary 
contributor to increased enrollment in the first year of college. 

• The scholarships cost additional money but do not increase the average cost per degree. After 
five years of follow-up, the cost per degree is about the same for students in the program group 
and the control group. 

Although the Opening Doors performance-based scholarship program in Louisiana produced size- 
able impacts on enrollment in several terms after random assignment, these large impacts were not 
found in the PBS Demonstration sites. In part, this outcome may be due to high rates of persistence 
among students in the control groups of the PBS Demonstration sites, yielding a bar that is difficult 
to surpass, or it may have occurred for other reasons. The economic contexts, for example, were also 
different, which may have given students different work incentives and options: The Louisiana program 
took place during an economic boom (2004 to 2005), but the PBS programs began during a period 
of economic downturn (2008 to 2010). 57 Many of the programs studied in the PBS Demonstration 
were in states that had more generous financial aid options for low-income students compared with 
Louisiana at the time of the Opening Doors study, potentially making the scholarship more valuable 
in Louisiana. 58 

While the long-term impact findings in this report are modest, the amounts of performance-based 
scholarships that students earned were also modest relative to students’ overall financial aid packages. 
For example, students received an average of $765 in performance-based scholarships at the Ohio 
colleges during the program year, compared with $7,947 in total financial aid in the same year. At 
the University of New Mexico, students received an average of $2,597 in performance-based schol- 
arships, compared with $20,487 in average total financial aid received during the program. These 
dollar amounts account for less and less of students’ total financial aid, as they continue in college 
after the programs end. Yet these scholarships still improved academic outcomes in the long term. 
These results are also promising in the context of existing research on financial aid, which generally 
finds small positive impacts or no evidence of impacts for enrollment or credit accumulation, and 
less frequently finds impacts on graduation. 59 


57. In 2004 the unemployment rate was 5.1 percent, and in 2005 it was 5.5 percent. From 2008 to 2010, the 
unemployment rate increased from 5.8 percent to 9.6 percent (Bureau of Labor Statistics, 2015). 

58. For example, in the 2004-2005 academic year, Louisiana provided an estimated $8.17 per student in need-based 
grants to undergraduate students (NASSGAP, 36th Annual NASSGAP Survey Report, n.d.), while in the 2008-2009 
academic year, estimated need-based grants ranged from $54.20 per undergraduate student to nearly $980 per 
undergraduate student in the six states in the PBS Demonstration (NASSGAP, 40th Annual NASSGAP Survey Report, 
n.d.). In the 2009-2010 academic year, estimated need-based grants ranged from $30.84 per undergraduate student 
to $1,027.23 per undergraduate student in those states (NASSGAP, 41st Annual NASSGAP Survey Report, n.d.). 

59. See, for example, Angrist, Lang, and Oreopoulos (2009); Binder, Krause, Miller, and Cerna (2015); Goldrick-Rab, 

Harris, Kelchen, and Benson (2012); and Scott-Clayton (2011). Ford, Grekou, Kwakye, and Nicholson (2014) provide an 
example of a related program that shows evidence of graduation impacts in Canada. 
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Lessons from the Demonstration 


This evaluation suggests that the performance-based scholarship programs accomplished the two 
main goals for the demonstration: helping low-income students both pay for college and make greater 
academic progress toward their degrees. A key lesson for scholarship providers and policymakers 
at the federal, state, and local levels is that these programs did not depend on rigid adherence to a 
particular scholarship structure, but instead were guided by three core principles: 

1. Scholarship payments made during a semester were associated with short-term academic be- 
haviors, such as enrollment in a certain number of credits and maintaining that course load 
over the semester. 

2. Scholarship payments at the end of a semester were tied to important benchmarks necessary for 
graduation (such as receiving a minimum grade in a certain number of credits over the semester), 
while providing funds to support students’ education in the next semester. 

3. When program designers perceived that their targeted student population would benefit from 
additional student services (such as advising or tutoring), the programs required that students 
use these services in order to earn the scholarships. This requirement improved the participa- 
tion in these services substantially. 

The program design in each state was created collaboratively with key stakeholders. These stakehold- 
ers provided valuable insight related to the target populations for the scholarships, the benchmarks, 
and the student services, if any, that should be incorporated into the design. The flexible nature of 
the programs makes it possible for any scholarship provider — at the state or local level, public or 
private — to adapt the design to its own priorities. 60 


Implications 

Several states have used existing funds in creative ways to encourage greater academic progress 
through college for low-income populations. Both the original performance-based scholarship 
program launched in Louisiana as part of the Opening Doors Demonstration and the Ohio PBS 
program described in this report used state Temporary Assistance for Needy Families (TANF) funds 
to offer these scholarships to low-income parents who were attending college. More recently, new 
legislation in Indiana includes provisions to give financial bonuses to some low-income students if 
they meet established academic benchmarks. Students in the Frank O’Bannon program, for example 
— a need-based financial aid program — can earn an additional $1,300 in aid if they earn 39 cred- 
its annually. 61 The California state legislature is currently considering a similar program. 62 States 
seeking to improve outcomes for low-income students may find similar opportunities in existing 


60. For more information, see Welbeck, Ware, Cerna, and Valenzuela (2014) on how performance-based scholarships 
can be adapted to local contexts and implemented. 

61. Indiana Commission for Higher Education (2015). 

62. The California state senate bill is SB 15. See California State Senate Majority Caucus (2014). 
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funds, and they can look to performance-based scholarships as an important tool in their efforts to 
improve graduation rates. 

Despite the positive findings of these programs, however, this study does not suggest that all financial 
aid should be restructured to promote performance — especially in regard to the federal Pell Grant 
program. In some ways, the Pell Grant program is already tied to performance; students remain 
eligible for their Pell Grants by meeting Satisfactory Academic Progress (SAP) requirements. The 
exact criteria vary by institution, but in most cases SAP means maintaining a grade point average 
of at least 2.0. 

In addition, the Pell Grant is generally the foundation of a student’s aid package and is much larger 
than the scholarship programs studied here. Performance-based scholarships in this study are also 
paid on top of Pell dollars and any other financial aid. Changing disbursement criteria for Pell could 
alter behavior in ways very different from the PBS Demonstration results. It could also negatively 
affect enrollment, since Pell is typically paid at the beginning of the semester and students may an- 
ticipate having those funds at the time they enroll in school. Providing students with less financial 
aid up front could reduce their ability to cover tuition and fees at many colleges. 63 

State and private donors contribute more than $20.1 billion in scholarships to undergraduate stu- 
dents, however, and some of this aid may be more effectively restructured as performance-based 
scholarships. 64 For the 2012-2013 academic year, 19 percent of all state-sponsored aid was reported 
as merit-based — typically for students’ high school performance, not their college performance. 65 
Students who receive this aid, however, may be likely to succeed in college without it. 66 Reallocating 
merit-based aid to low-income students through performance-based scholarship programs could 
be an effective strategy to help these students make greater academic progress. It could also help 
programs offer money to more students or increase the size of scholarships; students in the programs 
were offered the opportunity to earn more scholarships, but not all students did, so the programs 
did not pay out the full amount that was offered. Identifying specific students who will benefit from 
a scholarship program is challenging, but performance-based scholarships are designed to reach 
more of the students likely to benefit, because they are structured to pay only students who meet 
academic benchmarks during the program. Consequently, performance-based scholarships provide 
an opportunity to offer aid to more students without increasing the total amount of aid distributed. 
The evidence suggests that the cost per degree also does not increase, even after accounting for 
administrative costs. 


63. In 2010-2011, the median tuition and fees for all public institutions was $4,632. For all public four-year institutions it 
was $6,780, and for all public two-year institutions it was $2,537. These figures represent tuition and fees over a full 
year. See National Center for Education Statistics (2011). 

64. Baum, Cardenas Elliott, and Ma (2014). 

65. NASSGAP, 44th Annual NASSGAP Survey Report (n.d.). 

66. Baum et al. (2012). 
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Looking Ahead 

While the PBS Demonstration answered a number of significant questions regarding the efficacy 
of performance-based scholarships, some remain. Key among these is identifying the best scholar- 
ship structure to influence student outcomes. As discussed earlier, there is little evidence that the 
impacts from the PBS programs varied across states, even though the program differed in each state. 
There is also little evidence that the impacts from the programs varied across different subgroups. 

These results raise two crucial questions: Why not provide the least expensive scholarship option 
studied here — for example, the scholarship offered at the Ohio colleges? And, should additional 
services be included? The answers are not simple. Although the scholarship amounts varied across 
the programs, so did the costs of college. The students, their academic contexts, and the larger en- 
vironments also varied. The colleges and organizations involved in the studies played a large role in 
structuring these scholarships to meet the specific needs of student populations that they believed 
could benefit from performance-based scholarship programs. These individuals knew the student 
populations well, researched the existing academic outcomes and obstacles for these populations, and 
thoughtfully structured scholarships based on that information and existing services at the colleges. 
Differently designed scholarships may have different impacts for the same students in the same col- 
leges. The PBS Demonstration as a whole, moreover, was designed to evaluate whether performance- 
based scholarships could be effective in different conditions and for different student populations. 
It was not intended to identify the ideal structure of a performance-based scholarship program or 
to disentangle the effectiveness of individual program components, which remain open questions. 

The results presented here demonstrate that performance-based scholarship programs have helped 
students make greater academic progress than they would have without the programs. The programs 
provide a flexible design that can be implemented widely, and they can be structured to meet the 
needs of diverse student populations. The programs studied here consistently improved students’ 
academic progress. This evaluation shows that performance-based scholarships are an important 
tool for colleges, states, and scholarship providers who work to improve outcomes for low-income 
students. 
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APPENDIX 

A 


Technical Details for the Pooled 
and Site-Level Estimates 



This technical appendix provides additional information about the statistical analyses used in this report. 


DESIGN 

Multiple colleges participated in this study, and several colleges had multiple campuses. In California, 
students were recruited from four geographic areas throughout the state but could attend any college. 1 
Multiple cohorts of students were also recruited into the study. A cohort refers to a group of students 
entering at the beginning of a specific semester (for example, Borough of Manhattan Community Col- 
lege had a fall 2008 cohort, spring 2009 cohort, and fall 2009 cohort). In total, there were 45 unique 
cohort combinations by college/campus and geographic region. Individual students were randomly 
assigned to the program or control groups, and this random assignment was conducted separately for 
each unique combination of cohorts and campuses, or cohorts and geographic regions. 


WEIGHTING 

Weights were used in the pooled impact model so that each state contributed equally to the impact 
estimates. The weights account for the different sample sizes across states and for the varying random 
assignment ratios within each state. Weights were created for program and control group students in 
each state by calculating the program and control group ratios across all states and dividing these by 
the total number of states (six). These values were then divided by the percentage of program and con- 
trol group students in each state. 

As described in Table 2 in the main report, for the outcomes in this report, different follow-up periods 
are available for the states included. For some of the outcomes, data are available only for Years 1 
through 3 for some of the states, while data are available through Year 4 or Year 5 for other states. A 
different set of weights was calculated for the states included in each pooled outcome. 

Equation A. 1 shows an example of the formula that was used to calculate the weights. In this example, 
the weights for program group students in one state are calculated. Equivalent calculations were done 
for control group students in each state. 


Equation A.1 



where: 


'See Figure 1 in Ware and Patel (2012), 5. 
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Weight psm = 
N m = 

Nps 


V 


Npsm 


Nsm 


the weight for students in the program group ( ps ) in state m 
the count of states (m) included in the outcome 

the count of students in the program groups ( ps ) across all states in- 
cluded in the outcome 

the count of total students (s) across all states included in the outcome 
the count of students in the program group ( ps ) in state m 
the count of total students (s) in state m 


THE POOLED IMPACT MODEL 

The pooled model produces estimates of the average effect of the opportunity to participate in the per- 
formance-based scholarship programs. The equation in the model specifies an outcome measure (for 
example, total credits earned) as a function of indicator variables that identify each campus-cohort 
combination and a single treatment indicator that distinguishes between sample members randomly 
assigned to the program group and control group. 

As noted above, random assignment was conducted separately for each cohort of students at each 
campus or geographic region within the study. The campus-cohort indicators identify each unique 
campus-cohort combination. In total, there are 45 campus-cohort dummies. 


Equation A.2 


y = 


a kl Campus_ 

k 1 


x_Cohort kl + (3 0 t + e 


where: 


y = an outcome for a student 

Campus _x_Cohort k i = a campus-cohort dummy indicator equal to 1 if the student is 

from campus k and cohort l, and 0 otherwise 2 

t = a treatment indicator equal to 1 if the student was randomly 

assigned to the program group and 0 otherwise 


2 There are 45 unique campus-cohort dummy variables. 
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The estimated value of the treatment indicator coefficient (/ 5 0 ) is the estimated average effect of the 
opportunity to earn a performance-based scholarship. Tables and figures in this report present the 
least squares means for students in the program and control groups. 3 

Sensitivity analyses were conducted using cluster-robust standard errors at the campus-cohort level to 
check whether estimates and variance could be more similar within these groups than across the 
pooled sample. If clustering were an issue, it could result in smaller standard errors for the pooled im- 
pact estimates and lower p-values. 4 These checks were done for each of the primary pooled outcomes 
to determine whether there was a significant difference in the findings when the adjustments were 
made. Appendix Tables A.13, A. 14, and A. 15 show the results of those analyses. The adjustments for 
clustering do not affect the findings presented under “Impacts on Financial Aid and Student Out- 
comes” in this report. 


ESTIMATING VARIATION IN IMPACTS 

The previous section described the model used to obtain the pooled program effect. Described here 
are the models used to determine whether there is evidence of variation in impact estimates across 
states. 


Impact Variation by State 

Equation A. 3 describes a model that produces estimates of the average effect of the opportunity to 
earn a performance-based scholarship in each state. 


Equation A.3 


y 


= 11 

l k 


a k iCampus_x_Cohort k i + 




(t * State m ) + e 


where: 


State m = a dummy indicator equal to 1 if the student was randomly assigned in 

state m, and 0 otherwise, where there are six states 


The estimated value of the treatment indicator coefficient (/? m ) is the estimated average effect of the 
opportunity to earn a performance-based scholarship for state m’s sample. In order to test for impact 
variation among the states, a joint F-test is used. The null hypothesis is that each /? m is the same. More 
formally: 


Tor more details, refer to SAS Institute Inc. (2010). 

4 Cameron and Miller (2015). 
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H 0 : Pi — @2 — P'3 — — Ps ~ P(> 


If the null hypothesis is rejected, it means that there is evidence that the impacts vary across the 
states . 5 


5 The joint F-test is essentially testing to determine whether any predictive power is gained by including indicators 
in the model that identify participants' state and treatment condition, compared with just including a single 
treatment indicator, as in Equation A. 2. 
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APPENDIX TABLE A.l Selected Characteristics of 
Sample Members at Baseline, by Program 

Final Report on the Performance-Based Scholarship Demonstration 


Characteristic 

Arizona California 

Florida New Mexico New York 

Ohio 

Gender (%) 







Male 

100.0 

39.8 

33.7 

39.2 

30.9 

13.6 

Female 3 

0.0 

60.2 

66.3 

60.8 

69.1 

86.4 

Average age (years) 

24.3 

17.6 

27.0 

18.0 

26.5 

29.9 

Race/ethnicity b (%) 







Hispanic/Latino 

99.7 

60.7 

30.7 

60.6 

44.3 

8.6 

White 

0.1 

20.3 

30.5 

21.8 

6.1 

54.6 

Black 

0.0 

3.8 

33.1 

2.7 

37.2 

31.4 

Asian or Pacific Islander 

0.0 

11.2 

1.2 

3.5 

9.7 

0.6 

Other 

0.2 

3.9 

4.4 

11.3 

2.7 

4.8 

Flave any children (%) 

24.6 

NA 

44.1 

1.8 

47.8 

99.7 

Flousehold receiving any government benefits' 1 (%) 

37.5 

NA 

38.0 

NA 

37.9 

65.8 

Missing 

12.4 

NA 

12.6 

NA 

12.0 

4.0 

Currently employed (%) 

44.9 

NA 

50.8 

49.0 

56.0 

48.8 

First person in family to attend college (%) 

36.7 

53.7 

33.3 

32.8 

32.9 

30.0 

Sample size (total = 1 1,613) 

1,028 

4,642 

1,075 

1,081 

1,502 

2,285 


SOURCE: MDRC calculations using Baseline Information Form (BIF) data. 

NOTES: NA = Not available because question was not asked on BIF for this site. 

Missing values are included only in variable distributions for characteristics with more than 5 percent of the sample 
missing. 

a Female students were not eligible to participate in the program at Pima Community College; thus, gender was imputed 
and not explicitly asked on the BIF. 

Respondents who said they are FHispanic and chose a race are included only in the Flispanic category. Respondents who 
said they are not Flispanic and chose more than one race are included in the Other category. These respondents, combined 
with those who said they were American Indian or Alaska Native or another race/ethnicity, are included in the Other 
category. 

benefits include unemployment/dislocated worker benefits, Supplemental Security Income (SSI) or disability benefits, 
cash assistance or welfare, food stamps, and Section 8 or public housing. 
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APPENDIX TABLE A.2 Impacts on Enrollment in Years 1 -5, by State 







Final Report on the Performance-Based Scholarship Demonstration 





Arizona 

California 

Florida 

New Mexico 

New York 

Ohio 



Control 

Impact 

Control 

Impact 

Control 

Impact 

Control 

Impact 

Control 

Impact 

Control 

Impact 

F-Test 

Year 

Group (%) 

Estimate 3 

Group (%) 

Estimate 3 

Group (%) 

Estimate 3 

Group (%) 

Estimate 3 

Group (%) 

Estimate 3 

Group (%) 

Estimate 3 

P-Value 

1 

98.08 

0.78 

88.88 

3.91 *** 

96.75 

0.89 

100.00 

-0.94 

97.73 

1.21 

96.18 

1.04 

0.01 



0.38) 


(0.7) 


(1.37) 


(1.32) 


(1.12) 


(0.93) 


2 

68.19 

1.31 

82.54 

3.05 ** 

70.05 

0.60 

90.45 

-0.14 

74.74 

2.70 

71.11 

1.47 

0.85 



(2.56) 


(1.3) 


(2.54) 


(2.45) 


(2.08) 


(1.72) 


3 

49.93 

3.07 

76.44 

1.45 

56.53 

-4.41 

82.19 

2.15 

60.29 

3.77 

57.46 

-1.54 

0.18 



(2.88) 


(1.46) 


(2.86) 


(2.76) 


(2.34) 


(1.93) 


4 

- 

- 

71.11 

2.88 * 

- 

- 

77.98 

-0.55 

49.48 

2.23 

41.90 

1.38 

0.73 





(1.49) 




(2.81) 


(2.38) 


(1.97) 


5 

- 

- 

- 

- 

- 

- 

66.23 

-1.67 

34.77 

5.26 ** 

34.01 

0.73 

0.16 









(2.91) 


(2.46) 


(2.04) 


Sample size 

1,028 


4,642 


1,075 


1,081 


1,502 


2,285 



SOURCES: MDRC calculations using transcript data from the National Student Clearinghouse, Pima Community College, Hillsborough Community College, the University of 
New Mexico, the City University of New York, and the Ohio Board of Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; *=10 percent. 
A joint F-test was used to assess impact variation across states. The p-value for each test is shown. 

Standard errors are shown in parentheses below each estimate. 

Estimates are adjusted by site and research cohort. 

Estimates may vary slightly from state-level reports due to the exclusion of state-specific covariates. 

Enrollment measures whether a student enrolled at any point in a given year. 

a The impact estimate is the estimated change in outcomes caused by the program, measured by the difference between the program and control group outcomes. 
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APPENDIX TABLE A.3 Impacts on Total Credits Earned by the End of Years 1-4, by State 
Final Report on the Performance-Based Scholarship Demonstration 


Year 

Arizona 


Florida 

New Mexico 

New York 


Ohio 


Control 
Group (%) 

Impact 

Estimate 3 

Control 
Group (%) 

Impact 

Estimate 3 

Control 
Group (%) 

Impact 

Estimate 3 

Control 
Group (%) 

Impact 

Estimate 3 

Control 
Group (%) 

Impact 

Estimate 3 

F-Test 

P-Value 

1 

14.49 

1.81 *** 

13.95 

1.00 

25.19 

0.79 

15.89 

0.89 * 

13.94 

1 .69 *** 

0.53 



(0.64) 


(0.64) 


(0.61) 


(0.53) 


(0.43) 


2 

21.87 

2.61 ** 

22.12 

1.24 

45.32 

1.96 * 

26.54 

1.24 

22.37 

2.38 *** 

0.80 



(1.19) 


(1.18) 


(1.13) 


(0.97) 


(0.8) 


3 

25.55 

2.68 

27.81 

0.24 

63.48 

1.90 

34.06 

1.91 

28.43 

2.40 ** 

0.82 



(1.65) 


(1.63) 


(1.58) 


(1.35) 


(1.1) 


4 

- 

- 

- 

- 

79.91 

1.94 

39.42 

1.95 

32.20 

2.32 

0.98 







(2.06) 


(1.77) 


(1.45) 


Sample size 

1,028 


1,075 


1,081 


1,469 


2,285 



SOURCES: MDRC calculations using transcript data from Pima Community College, Hillsborough Community College, the University of New Mexico, the City 
University of New York, and the Ohio Board of Regents. 


NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; 
* = 10 percent. 

A joint F-test was used to assess impact variation across states. The p-valuefor each test is shown. 

Standard errors are shown in parentheses below each estimate. 

Estimates are adjusted by site and research cohort. 

Estimates may vary slightly from state-level reports due to the exclusion of state-specific covariates. 

a The impact estimate is the estimated change in outcomes caused by the program, measured by the difference between the program and control group 
outcomes. 


in 


APPENDIX TABLE A.4 Impacts on Cumulative Credits 
Attempted, Pooled 

Final Report on the Performance-Based Scholarship Demonstration 


Cumulative 
Credits Attempted 


Year 3 

Sample 

Size 

Program 

Group 

Control 

Group 

Impact 

Estimate 13 

Standard 

Error 

P-Value 

1 

6,938 

23.9 

23.0 

q g *** 

0.21 

0.00 

2 

6,938 

39.5 

37.8 

1 ~j *** 

0.45 

0.00 

3 

6,938 

50.1 

48.4 

1 -j *** 

0.65 

0.01 

4 

4,835 

63.4 

61.4 

1.9 * 

1.08 

0.07 

5 

1,081 

109.3 

109.3 

0.0 

3.38 

0.99 


SOURCES: MDRC calculations using transcript data from Pima Community College, Hillsborough 
Community College, the University of New Mexico, the City University of New York, and the Ohio 
Board of Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical 
significance levels are indicated as: *** = 1 percent;** = 5 percent;* = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

Estimates are weighted to account for the different sample sizes and random assignment ratios in 
each state. 

a Years 1, 2, and 3 each include Arizona, Florida, New Mexico, New York, and Ohio. Year 4 includes 
New Mexico, New York, and Ohio. Year 5 includes New Mexico. 

b The impact estimate is the estimated change in outcomes caused by the program, measured by 
the difference between the program and control group outcomes. 
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APPENDIX TABLE A.5 Impacts on Degrees Earned by the End of Years 1-5, by State 






Final Report on the Performance-Based Scholarship Demonstration 





Arizona 

California 

Florida 

New Mexico 

New York 

Ohio 



Control 

Impact 

Control 

Impact 

Control 

Impact 

Control 

Impact 

Control 

Impact 

Control 

Impact 

F-Test 

Year 

Group (%) 

Estimate 3 

Group (%) 

Estimate 3 

Group (%) 

Estimate 3 

Group (%) 

Estimate 3 

Group (%) 

Estimate 3 

Group (%) Estimate 3 

P-Value 

1 

4.86 

-0.32 

0.55 

0.04 

0.24 

1.10 

0.37 

-0.37 

2.94 

-0.55 

8.47 

1.06 

0.65 



(1.04) 


(0.53) 


(1.03) 


(0.99) 


(0.84) 


(0.7) 


2 

11.38 

1.47 

2.93 

-0.14 

4.47 

2.81 * 

0.73 

-0.36 

10.56 

-1.54 

17.40 

3.20 *** 

0.06 



(1.66) 


(0.84) 


(1.65) 


(1.59) 


(1.35) 


(1.11) 


3 

19.79 

1.40 

7.68 

-0.56 

13.93 

3.90 * 

2.57 

-0.33 

19.65 

-0.03 

24.73 

3.75 *** 

0.17 



(2.16) 


(1.09) 


(2.14) 


(2.07) 


(1.75) 


(1.45) 


4 

- 

- 

22.95 

0.40 

- 

- 

17.61 

1.23 

28.88 

-0.62 

33.15 

1.80 

0.88 





(1.4) 




(2.64) 


(2.24) 


(1.85) 


5 

- 

- 

- 

- 

- 

- 

40.74 

4.41 

31.68 

2.41 

34.76 

2.99 

0.86 









(2.92) 


(2.47) 


(2.04) 


Sample size 

1,028 


4,642 


1,075 


1,081 


1,502 


2,285 



SOURCES: MDRC calculations using transcript data from the National Student Clearinghouse, Pima Community College, Hillsborough Community College, the University 
of New Mexico, the City University of New York, and the Ohio Board of Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; 

*=10 percent. 

A joint F-test was used to assess impact variation across states. The p-value for each test is shown. 

Standard errors are shown in parentheses below each estimate. 

Estimates are adjusted by site and research cohort. 

Estimates may vary slightly from state-level reports due to the exclusion of state-specific covariates. 

a The impact estimate is the estimated change in outcomes caused by the program, measured by the difference between the program and control group outcomes. 


APPENDIX TABLE A.6 Services Versus No Services, by Registration, 
Credits Earned, and Degrees Earned, Years 1-3 

Final Report on the Performance-Based Scholarship Demonstration 

Subgroup 

Sample 

Size 

Percentage 
Program Control 

Group Group 

Impact 

Estimate 3 

Standard 

Error 

Difference Between 
Subgroup Impacts 
(P-Value) 

Year 1 

Registration 

Services 

3,184 

98.5 

98.3 

0.2 

0.4 

0.01 ttt 

No services 

8,429 

94.8 

92.7 

2 i *** 

0.5 


Credits earned 
Services 

3,184 

19.1 

17.9 

"1 2 *** 

0.4 

0.84 

No services 

3,754 

16.0 

14.8 

1.3 *** 

0.3 


Degrees earned 
Services 

3,184 

1.9 

1.8 

0.1 

0.5 

0.95 

No services 

8,429 

3.3 

3.1 

0.2 

0.4 


Year 2 

Registration 

Services 

3,184 

77.0 

76.4 

0.6 

1.5 

0.29 

No services 

8,429 

80.5 

78.1 

2.4 *** 

0.9 


Credits earned 
Services 

3,184 

31.8 

29.9 

1 g *** 

0.7 

0.89 

No services 

3,754 

25.9 

24.1 

1 .8 *** 

0.6 


Degrees earned 
Services 

3,184 

6.7 

5.4 

1.3 

0.8 

0.44 

No services 

8,429 

8.7 

8.2 

0.5 

0.7 


Year 3 

Registration 

Services 

3,184 

63.4 

63.1 

0.3 

1.6 

0.63 

No services 

8,429 

69.6 

68.3 

1.2 

1.0 


Credits earned 
Services 

3,184 

40.8 

39.2 

1.6 * 

1.0 

0.66 

No services 

3,754 

32.8 

30.7 

2 2 *** 

0.8 


Degrees earned 
Services 

3,184 

13.6 

12.0 

1.7 

1.2 

0.67 

No services 

8,429 

15.3 

14.3 

1.0 

0.8 



(continued) 
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APPENDIX TABLE A.6 (continued) 


SOURCES: For "Registration" and "Degrees earned," "Services" is based on transcript data from the National Student 
Clearinghouse, Arizona, Florida, and New Mexico, and "No services" is based on transcript data from the National 
Student Clearinghouse, New York, and Ohio. For "Credits earned," "Services" is based on transcript data from Arizona, 
Florida, and New Mexico, and "No services" is based on transcript data from New York and Ohio. 

NOTES: A two-tailed t-test was applied to differences in impacts between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; *=10 percent. 

A two-tailed t-test was applied to differences in impacts between subgroups. Statistical significance levels are 
indicated as: ttt = 1 percent; ft = 5 percent; t = 1 0 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

Estimates are weighted to account for the different sample sizes and random assignment ratios in each state. 
Enrollment measures whether a student enrolled at any point in a given year. 

a The impact estimate is the estimated change in outcomes caused by the program, measured by the difference 
between the program and control group outcomes. 
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APPENDIX TABLE A.7 Impacts on Enrollment, by California Scholarship Type 
Final Report on the Performance-Based Scholarship Demonstration 

Outcome 

Control 

Group 

Mean 



Scholarship Type 




F-Test 

P-Value 

1 

2 

3 

4 

5 

6 

Performance-based scholarship 


No 

Yes 

Yes 

Yes 

Yes 

Yes 


Amount of scholarship per semester ($) 


1,000 

1,000 

500 

1,000 

500 

1,000 


Duration of scholarship 


1 semester 

1 semester 

2 semesters 2 semesters 4 semesters 

4 semesters 





Enrollment Impact over Control Group Mean (%) 




Year 









1 

88.9 

3.7 ** 

2.7 

3.9 ** 

3.9 ** 

^ 1 *** 

4.0 ** 

1.0 



(1.8) 

(1.9) 

(1.9) 

(1.8) 

(1.8) 

(1.9) 


2 

82.5 

1.8 

2.9 

4.4 * 

1.2 

4.1 * 

2.6 

0.9 



(2.3) 

(2.3) 

(2.3) 

(2.3) 

(2.3) 

(2.3) 


3 

76.4 

-3.2 

2.6 

-0.5 

1.5 

2.6 

1.0 

0.5 



(2.6) 

(2.7) 

(2.6) 

(2.6) 

(2.6) 

(2.6) 


4 

71.1 

-0.7 

1.0 

1.6 

3.3 

4.0 

4.5 

0.7 



(2.8) 

(2.8) 

(2.8) 

(2.8) 

(2.8) 

(2.8) 


Sample size (n = 4,921) 

3,281 

279 

264 

273 

276 

276 

272 
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APPENDIX TABLE A.7 (continued) 


SOURCE: MDRC calculations using National Student Clearinghouse data. 

NOTES: A joint F-test was used to assess impact variation across programs. The p-value for each test is shown. For each of the program years, the test yielded a p-value on 
the F-statistic that was not significant. This finding suggests that the differences in each program year are likely to have occurred by chance. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; *=10 percent. 
Rounding may cause slight discrepancies in sums and differences. 

Impact estimates are adjusted by site and research cohort. 

Standard errors are reported in parentheses under impact estimates. 

National Student Clearinghouse data were not found for 228 students (4.6 percent of the sample). 

The PBS California scholarship types had varying designs. Type 1 was not performance-based and offered $1,000 over one term with no performance incentive. Types 2-6 
were performance-based as follows: Type 2 offered $1,000 over one term; Type 3 offered $1,000 over one year; Type 4 offered $2,000 over one year; Type 5 offered $2,000 
over two years; and Type 6 offered $4,000 over two years. For more details see Richburg-Flayes et al. (201 5). 
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APPENDIX TABLE A.8 Impacts on Degrees Earned, by California Scholarship Type 
Final Report on the Performance-Based Scholarship Demonstration 

Control 

Group 

Outcome Mean 



Scholarship Type 




F-Test 

P-Value 

1 

2 

3 

4 

5 

6 

Performance-based scholarship 

No 

Yes 

Yes 

Yes 

Yes 

Yes 


Amount of scholarship per semester ($) 

1,000 

1,000 

500 

1,000 

500 

1,000 


Duration of scholarship 

1 semester 

1 semester 

2 semesters 2 semesters 4 semesters 4 semesters 




Degree Impact over Control Group Mean (%) 




Year 








1 0.5 

-0.5 

0.2 

0.2 

-0.5 

-0.2 

0.5 

0.4 


(0.5) 

(0.5) 

(0.5) 

(0.5) 

(0.5) 

(0.5) 


2 2.9 

-0.1 

0.1 

0.4 

-0.4 

-1.8 * 

1.1 

0.4 


(1.0) 

(1.1) 

(1.1) 

(1.0) 

(1.0) 

(1.1) 


3 7.7 

0.2 

-1.6 

1.1 

-1.9 

-1.8 

1.5 

0.4 


(1.6) 

(1.7) 

(1.7) 

(1.6) 

(1.6) 

(1.7) 


4 23.0 

-3.9 

-0.3 

1.3 

-2.3 

2.9 

0.4 

0.4 


(2.6) 

(2.7) 

(2.6) 

(2.6) 

(2.6) 

(2.6) 


Sample size (n = 4,921) 3,281 

279 

264 

273 

276 

276 

272 
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APPENDIX TABLE A.8 (continued) 


SOURCE: MDRC calculations using National Student Clearinghouse data. 

NOTES: A joint F-test was used to assess impact variation across programs. The p-value for each test is shown. For each of the program years, the test yielded a p-value 
on the F-statisticthat was not significant. This finding suggests that the differences in each program year are likely to have occurred by chance. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; *=10 percent. 
Rounding may cause slight discrepancies in sums and differences. 

Impact estimates are adjusted by site and research cohort. 

Standard errors are reported in parentheses under impact estimates. 

National Student Clearinghouse data were not found for 228 students (4.6 percent of the sample). 

The PBS California scholarship types had varying designs. Type 1 was not performance-based and offered $1,000 over one term with no performance incentive. Types 
2-6 were performance-based as follows: Type 2 offered $1,000 over one term; Type 3 offered $1,000 over one year; Type 4 offered $2,000 over one year; Type 5 offered 
$2,000 over two years; and Type 6 offered $4,000 over two years. For more details see Richburg-Flayes et al. (2015). 
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APPENDIX TABLE A.9 Impacts on Enrollment in Years 1-5, by Subgroup 
Final Report on the Performance-Based Scholarship Demonstration 


Characteristic 

Year 1 


Year 2 

Year 3 

Year 4 


Year 5 

Impact 

Estimate 3 

Standard 

Error 

Impact 

Estimate 3 

Standard 

Error 

Impact 

Estimate 3 

Standard 

Error 

Impact Standard 
Estimate 3 Error 

Impact 

Estimate 3 

Standard 

Error 

Gender 





ft 






Male 

1 .53 *** 

0.5 

2.45 * 

1.3 

3.19 ** 

1.4 

0.84 

1.7 

-0.63 

2.8 

Female 

0.89 ** 

0.4 

1.09 

0.9 

-0.85 

1.1 

1.60 

1.2 

1.88 

1.6 

Hispanic/Latino 











Yes 

1.54 *** 

0.5 

1.44 

1.1 

1.81 

1.2 

3.20 ** 

1.4 

1.72 

2.4 

No 

0.75 * 

0.4 

1.53 

1.1 

-0.25 

1.2 

0.09 

1.3 

1.23 

1.7 

Parent b 











Yes 

1 .07 ** 

0.5 

0.16 

1.5 

-0.97 

1.6 

1.88 

1.8 

2.06 

1.7 

No 

0.23 

0.4 

1.54 

1.4 

1.34 

1.6 

0.10 

2.1 

0.59 

2.3 

Younger than 20 years old 











Yes 

0.86 * 

0.5 

1.05 

0.9 

0.78 

1.1 

1.19 

1.1 

-1.80 

2.8 

No 

1 .40 *** 

0.4 

1.72 

1.3 

0.34 

1.4 

1.79 

1.7 

3.09 * 

1.6 

First in family to attend college 











Yes 

1 .66 *** 

0.6 

1.14 

1.2 

0.56 

1.4 

2.54 * 

1.5 

2.38 

2.5 

No 

0.80 ** 

0.4 

1.87 * 

1.0 

0.68 

1.1 

0.50 

1.3 

0.59 

1.7 

Employed at the 











beginning of the program b 







t 




Yes 

1 .06 ** 

0.5 

-0.20 

1.4 

0.02 

1.6 

3.31 * 

1.9 

3.33 * 

2.0 

No 

0.31 

0.5 

2.38 

1.5 

0.80 

1.6 

-1.64 

2.0 

-0.75 

2.0 


(continued) 


APPENDIX TABLE A.9 (continued) 


SOURCES: MDRC calculations using Baseline Information Form (BIF) data and transcript data from the National Student Clearinghouse, Pima Community College, 
Hillsborough Community College, the University of New Mexico, the City University of New York, and the Ohio Board of Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent;** = 5 percent;* = 10 percent. 

A two-tailed t-test was applied to differences in impacts between subgroups. Statistical significance levels are indicated as: ttt = 1 percent; tt = 5 percent; 
t = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

Estimates are weighted to account for the different sample sizes and random assignment ratios in each state. 

Years 1,2, and 3 each include Arizona, California, Florida, New Mexico, New York, and Ohio. Year 4 includes California, New Mexico, New York, and Ohio. Year 5 includes 
New Mexico, New York, and Ohio. 

a The impact estimate is the estimated change in outcomes caused by the program, measured by the difference between the program and control group outcomes. 
b Data on parental status and employment were not collected for California students. 
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APPENDIX TABLE A.io Impacts on Credits Earned in Years 1-4, by Subgroup 
Final Report on the Performance-Based Scholarship Demonstration 



Year 1 


Year 2 


Year 3 


Year 4 


Characteristic 

Impact 

Estimate 3 

Standard 

Error 

Impact 

Estimate 3 

Standard 

Error 

Impact 

Estimate 3 

Standard 

Error 

Impact 

Estimate 3 

Standard 

Error 

Gender 

Male 

t 

1 .77 *** 

0.4 

2 41 *** 

0.8 

2.31 ** 

1.0 

2.26 

2.3 

Female 

0.83 *** 

0.3 

1.54 *** 

0.6 

1.54 * 

0.8 

2.02 * 

1.2 

Fiispanic/Latino 

Yes 

1 .05 *** 

0.4 

1.57 ** 

0.7 

1.63 

1.0 

2.18 

2.1 

No 

1.45 *** 

0.3 

2.23 *** 

0.6 

2.03 ** 

0.8 

1.98 * 

1.2 

Parent 

Yes 

1 .20 *** 

0.3 

2.08 *** 

0.6 

2.32 *** 

0.8 

3.05 *** 

1.1 

No 

1 .20 *** 

0.4 

1 .55 ** 

0.7 

1.14 

1.0 

0.94 

2.0 

Younger than 20 years old 

Yes 

0.86 * 

0.5 

1.63 * 

1.0 

1.33 

1.4 

2.46 

2.7 

No 

1 .38 *** 

0.3 

1 95 *** 

0.5 

1 95 *** 

0.7 

1.82 * 

1.0 

First in family to attend college 

Yes 

1 .55 *** 

0.4 

2.27 *** 

0.8 

2.32 ** 

1.2 

3.14 

2.0 

No 

1.02 *** 

0.3 

1 .68 *** 

0.6 

1.51 * 

0.8 

1.23 

1.3 

Employed at the 
beginning of the program 

Yes 

1 .38 *** 

0.3 

2.02 *** 

0.6 

1 .97 ** 

0.9 

3.47 ** 

1.5 

No 

1.15 *** 

0.4 

1 .87 *** 

0.7 

1.79 * 

0.9 

0.48 

1.5 


(continued) 


APPENDIX TABLE A.io (continued) 


SOURCES: MDRC calculations using Baseline Information Form (BIF) data and transcript data from Pima Community College, Hillsborough Community College, the 
University of New Mexico, the City University of New York, and the Ohio Board of Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; 

*=10 percent. 

A two-tailed t-test was applied to differences in impacts between subgroups. Statistical significance levels are indicated as: ttt = 1 percent; ft = 5 percent; 
t = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

Estimates are weighted to account for the different sample sizes and random assignment ratios in each state. 

Years 1, 2, and 3 each include Arizona, Florida, New Mexico, New York, and Ohio. Year 4 includes New Mexico, New York, and Ohio. 

a The impact estimate is the estimated change in outcomes caused by the program, measured by the difference between the program and control group outcomes. 
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APPENDlXTABLEA.il Impacts on Degrees Earned in Years 1-5, by Subgroup 
Final Report on the Performance-Based Scholarship Demonstration 




Year 1 

Year 2 


Year 3 


Year 4 

Year 5 


Characteristic 

Impact 

Estimate 3 

Standard 

Error 

Impact Standard 
Estimate 3 Error 

Impact 

Estimate 3 

Standard 

Error 

Impact 

Estimate 3 

Standard 

Error 

Impact Standard 
Estimate 3 Error 

Gender 

Male 

0.24 

0.5 

1.37 * 

0.8 

0.96 

1.0 

-1.38 

1.4 

-0.46 

2.8 

Female 

0.07 

0.4 

0.56 

0.7 

1.60 * 

0.8 

1.43 

1.1 

4.56 *** 

1.6 

Hispanic/Latino 

Yes 

-0.29 

0.4 

t 

0.06 

0.7 

0.32 

0.9 

0.07 

1.3 

t 

0.23 

2.5 

No 

0.64 

0.5 

1 .78 ** 

0.8 

2.33 ** 

1.0 

1.23 

1.3 

5.21 *** 

1.7 

Parent 13 

Yes 

0.39 

0.8 

1.96 * 

1.1 

3.04 ** 

1.3 

1.81 

1.7 

3.24 * 

1.7 

No 

0.02 

0.5 

0.35 

0.9 

0.40 

1.2 

-0.32 

1.9 

3.17 

2.3 

Younger than 20 years old 
Yes 

-0.02 

0.2 

tt 

-0.53 

0.4 

tt 

-0.52 

0.6 

0.74 

1.1 

4.14 

2.9 

No 

0.30 

0.6 

2.06 ** 

0.9 

2.80 ** 

1.1 

0.75 

1.5 

2.85 * 

1.6 

First in family to attend college 

Yes 

-0.09 

0.5 

0.49 

0.8 

1.17 

1.0 

-0.20 

1.4 

3.21 

2.5 

No 

0.22 

0.4 

1.17 * 

0.7 

1.49 * 

0.9 

0.79 

1.2 

2.78 

1.7 

Employed at the 
beginning of the program 13 

Yes 

0.37 

0.6 

0.39 

1.0 

0.88 

1.3 

2.08 

1.8 

3.84 * 

2.0 

No 

0.00 

0.6 

1.50 

0.9 

2.41 ** 

1.2 

-0.27 

1.8 

3.10 

2.0 


(continued) 


APPENDlXTABLEA.il (continued) 


SOURCES: MDRC calculations using Baseline Information Form (BIF) data and transcript data from the National Student Clearinghouse, Pima Community College, 
Fiillsborough Community College, the University of New Mexico, the City University of New York, and the Ohio Board of Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; *=10 percent. 

A two-tailed t-test was applied to differences in impacts between subgroups. Statistical significance levels are indicated as: t+t = 1 percent; tt = 5 percent; t = 1 0 
percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

Estimates are weighted to account for the different sample sizes and random assignment ratios in each state. 

Years 1, 2, and 3 each include Arizona, California, Florida, New Mexico, New York, and Ohio. Year 4 includes California, New Mexico, New York, and Ohio. Year 5 includes 
New Mexico, New York, and Ohio. 

a The impact estimate is the estimated change in outcomes caused by the program, measured by the difference between the program and control group outcomes. 
b Data on parental status and employment were not collected for California students. 


APPENDIX TABLE A.12 Cost-Effectiveness of PBS Program 
Final Report on the Performance-Based Scholarship Demonstration 

Outcome 

Average 

Program 

Average 

Control 

Impact 

Estimate 3 

Cost of PBS program ($) 

2,345 

0 

2,345 

Total cumulative cost of college services 11 ' c ' d ($) 




Year 1 

9,160 

9,160 

0 

Year 2 

15,055 

15,055 

0 

Year 3 

1 9,242 

19,242 

0 

Year 4 

24,449 

24,449 

0 

Year 5 

27,809 

27,809 

0 

Earned a degree e,f ' s (%) 




Year 1 

2.9 

2.8 

0.2 

Year 2 

8.3 

7.4 

0.9 * 

Year 3 

14.9 

13.6 

1.4 ** 

Year 4 

26.4 

25.7 

0.7 

Year 5 

38.3 

35.1 

3.3 ** 

Cost per degree e,f ' 9,h ($) 




Year 1 

392,884 

330,934 

61,950 

Year 2 

210,260 

204,265 

5,996 

Year 3 

144,542 

141,670 

2,873 

Year 4 

101,420 

95,047 

6,373 

Year 5 

78,672 

79,310 

-638 


SOURCES: MDRC calculations using transcript data from the National Student Clearinghouse, Pima 
Community College, Hillsborough Community College, the University of New Mexico, the City University 
of New York, and the Ohio Board of Regents, as well as program-specific budget data and financial and 
enrollment data from the Integrated Postsecondary Education Data System and the Higher Education 
Price Index. 

NOTES: Program costs are based on a steady state of operation that excludes external research and start- 
up costs; the cost would increase by 0.7 percent if start-up costs were included and by 4.0 percent if 
research costs were included. 

All costs were adjusted into constant 2014 dollars. 

A two-tailed t-test was applied to differences between research groups for credits attempted and 
degrees earned. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent; 

* = 10 percent. Tests of statistical significance were not performed for costs. 
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APPENDIX TABLE A.12 (continued) 


Rounding may cause slight discrepancies in sums and differences. 

The average cost per credit attempted across all the sites over 5 years was $397.90. 
a The impact estimate is the estimated change in outcomes caused by the program, measured by the 
difference between the program and control group outcomes. 

b Years 1,2, and 3 are based on the average number of credits attempted for Arizona, Florida, New 
Mexico, New York, and Ohio (excludes California). Over three years, the average control group member 
across Arizona, Florida, New Mexico, New York, and Ohio attempted 48.4 credits. 

c Year 4 is based on the average number of credits attempted for New Mexico, New York, and Ohio 
(excludes Arizona, California, and Florida). Over 4 years, the average control group member across New 
Mexico, New York, and Ohio attempted 61 .4 credits. 

d Year 5 is based on an estimate of 69.9 credits attempted per control group member. This estimate is 
based on the average number of credits attempted for: New Mexico, New York, and Ohio over 4 years 
(61 .4), multiplied by the percentage increase in credits attempted from Year 4 to Year 5 as observed in New 
Mexico (13.7 percent). In Year 4, the typical control group member in New Mexico had attempted 96.1 
credits. By Year 5, the typical control group member in New Mexico had attempted 109.3 credits. The Year 
5 estimates are derived in this way because Year 5 data are available only for New Mexico. The University of 
New Mexico, however, offers 4-year degrees, and credit-attempt values were substantially higher 
compared with the other sites, so Year 5 estimates were derived from the Year 5 data from the University 
of New Mexico and the Year 4 data from the University of New Mexico, the New York colleges, and the 
Ohio colleges. 

e For Years 1,2, and 3, includes Arizona, California, Florida, New Mexico, New York, and Ohio. 
f For Year 4, includes California, New Mexico, New York, and Ohio. 

9For Year 5, includes New Mexico, New York, and Ohio. 

h The cost per degree for both students in the program groups and students in the control groups is 
calculated by adding the cost of usual college services to the direct cost of the average PBS program 
(program group only) and dividing by the estimated effect on earning a degree. For example, to calculate 
the 5-year cost per degree for the average program student, the 5-year cost of college services ($27,809) is 
added to the direct cost of the average PBS program ($2,345) for a total of $30,154 ($27,809 + $2,345 = 
$30,154). This total cost is then divided by the estimated percentage of program students who earned a 
degree after Year 5 (38.3 percent), for a cost per degree of $78,672 ($30,154 / [38.3/100] = $78,672). 
Discrepancies in calculations are due to rounding of impact estimates and costs. 
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APPENDIX TABLE A.13 Sensitivity Check, Clustered Standard Errors 
(Campus- and Geographic Region-Cohort), Impacts on Enrollment, Pooled 

Final Report on the Performance-Based Scholarship Demonstration 


Percentage Enrolled 


Year 3 

Sample 

Size 

Program 

Group 

Control 

Group 

Impact 

Estimate 13 

Standard 

Error 

P-Value 

1 

11,613 

95.6 

94.4 

1 2 *** 

0.42 

0.01 

2 

11,613 

79.3 

77.8 

1.5 ** 

0.70 

0.04 

3 

11,613 

67.8 

67.0 

0.8 

0.82 

0.36 

4 

9,510 

63.2 

61.7 

1.5 

0.99 

0.15 

5 

4,868 

42.9 

41.5 

1.4 

1.46 

0.34 


SOURCES: MDRC calculations using transcript data from the National Student Clearinghouse, Pima Community 
College, Hillsborough Community College, the University of New Mexico, the City University of New York, and 
Ohio Board of Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent;* = 10 percent. 

Enrollment measures whether a student enrolled at any point in a given year. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

Estimates were calculated using clustered standard errors by site and research cohort. 

Estimates are weighted to account for the different sample sizes and random assignment ratios in each 
state. 

a Years 1, 2, and 3 each include Arizona, California, Florida, New Mexico, New York, and Ohio. Year 4 includes 
California, New Mexico, New York, and Ohio. Year 5 includes New Mexico, New York, and Ohio. 

b The impact estimate is the estimated change in outcomes caused by the program, measured by the 
difference between the program and control group outcomes. 
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APPENDIX TABLE A.14 Sensitivity Check, Clustered Standard Errors 
(Cam pus- Cohort), Impacts on Cumulative Credits Earned, Pooled 

Final Report on the Performance-Based Scholarship Demonstration 


Cumulative 
Credits Earned 


Year 3 

Sample 

Size 

Program 

Group 

Control 

Group 

Impact 

Estimate 6 

Standard 

Error 

P-Value 

1 

6,938 

17.5 

16.2 

1 2 *** 

0.30 

0.00 

2 

6,938 

28.6 

26.7 

1 g *** 

0.57 

0.00 

3 

6,938 

36.4 

34.6 

1.8 ** 

0.68 

0.01 

4 

4,835 

47.2 

45.1 

2.1 * 

1.19 

0.10 


SOURCES: MDRC calculations using transcript data from Pima Community College, Hillsborough 
Community College, the University of New Mexico, the City University of New York, and the Ohio Board of 
Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent;** = 5 percent; *= 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

Estimates were calculated using clustered standard errors by site and research cohort. 

Estimates are weighted to account for the different sample sizes and random assignment ratios in each 
state. 

a Years 1, 2, and 3 each include Arizona, Florida, New Mexico, New York, and Ohio. Year 4 includes New 
Mexico, New York, and Ohio. 

b The impact estimate is the estimated change in outcomes caused by the program, measured by the 
difference between the program and control group outcomes. 
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APPENDIX TABLE A.15 Sensitivity Check, Clustered Standard Errors 
(Campus- and Geographic Region-Cohort), Impacts on Degrees 

Earned, Pooled 

Final Report on the Performance-Based Scholarship Demonstration 


Percentage Who 
Earned a Degree 


Year 3 

Sample 

Size 

Program 

Group 

Control 

Group 

Impact 

Estimate 6 

Standard 

Error 

P-Value 

1 

11,613 

2.9 

2.8 

0.2 

0.4 

0.65 

2 

11,613 

8.3 

7.4 

0.9 

0.6 

0.15 

3 

11,613 

14.9 

13.6 

1.4 * 

0.7 

0.05 

4 

9,510 

26.4 

25.7 

0.7 

0.7 

0.33 

5 

4,868 

38.3 

35.1 

2 3 *** 

0.8 

0.00 


SOURCES: MDRC calculations using transcript data from the National Student Clearinghouse, Pima 
Community College, Hillsborough Community College, the University of New Mexico, the City 
University of New York, and the Ohio Board of Regents. 

NOTES: A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as: *** = 1 percent; ** = 5 percent; *=10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Estimates are adjusted by site and research cohort. 

Estimates were calculated using clustered standard errors by site and research cohort. 

Estimates are weighted to account for the different sample sizes and random assignment ratios in 
each state. 

a Years 1,2, and 3 each include Arizona, California, Florida, New Mexico, New York, and Ohio. Year 4 
includes California, New Mexico, New York, and Ohio. Year 5 includes New Mexico, New York, and 
Ohio. 

b The impact estimate is the estimated change in outcomes caused by the program, measured by 
the difference between the program and control group outcomes. 
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